Teacher Preparation Experiences 
and Early 
Teaching Effectiveness 
PAN od od =A) LO =e 


NATIONAL CENTER ror 


& 
TOS INSMUTES EDUCATION EVALUATION 
= HG EDUCATION SCIENCES AND REGIONAL ASSISTANCE 


Contents 


Appendix A. Teacher Survey Response Rates by District/State...............scsscssscsssssseesssscsssseseesesssseseseeee 1 
Appendix B. Comparison of the Subsample Used for Relational Analyses and the Full Sample........ 3 
Appendix C. Development of the Teacher Survey .............scscsssscsssscsssssecesssscscscsesscsssessessessessseesessesssseneeee 7 
Appendix D. Teacher Survey Questions on Preparation Experiences..............ssscssscssssssssssseceessesseeees 11 
Appendix E. Measures of Preparation Experience .............ssssccsssssscssecsesscsscssscsscssssesecsesessessesceseessceees 27 
Appendix F. Teachers’ Assessment of the Usefulness of Preparation Experiences ..............cssesseseeee 29 
Appendix G. Factor Analysis Method ology................cssccsssssssscscesssssscesecsscssssssessessesssesseesssssesceseessesees 34 
Appendix H. Assessing Measurement Error in Teacher Survey Responses ............sssssssssessesesseseseees 37 
Appendix I. Value-Added Modeling Methodology ..................scsccssscssssssscsssssscsssccsscsescsesecssescecsseessnceenes 43 
Appendix J. Teachers’ Preparation Experiences by Preparation Provider and Degree Program ... 49 
Appendix K. Robustness of Results to Alternative Statistical Models .................sscscssssssseeseeseesessseees 52 
Appendix L. Subgroup Analyses for Relational Findings.................scsscsssssssssssssessessecesessesseseceserseses 55 


Appendix M. Relationships Between Preparation Experiences and Effectiveness by Year of 
TO aching .:scciscnissssseesisasieessnssesscscsescscsctesscsceieesosdéecsesssbeedsecssendteadteessesseesivassoessescdenseesocneséstessesestecsescisens 60 


NCEE 2019-4007 Teacher Preparation Experiences and Early Teaching Effectiveness i 


List of Figures 


Figure B.1: Percentages of Study Districts by Demographic Characteristics in the Relational and 


Full Samples, Weighted by Number of Teachers ............cssssssscsseceececeeeceseceeeeeceseceaeeaceaeeeeaecaeeeneeaeeneeeeeenees 4 
Figure B.2: Percentage of Study Teachers by Certification Route in the Relational and Full 

SaMMPleS 5 ccdsies coeees ssdeeecddanecddersackcekssedaed soeusae de sededl sscceuhésscedubbs sevesiessdudisaevedvsls etuddisvesdedistessessteditiaseeeddiaunennientds 5 
Figure B.3: Percentage of Study Teachers by Degree Program in the Relational and Full Samples............ 6 
Figure C.1: Steps in Constructing the Survey Questions on Teacher Preparation Experiences ..............+6+ 8 
Figure C.2: Round 1 Cognitive Testing Probes for Productive Use of Classroom Time .........:::ssscesceeseeees 9 


Figure E.1: Average Frequencies of Preparation Experiences Obtained About a Competency Area 
Through Four Types of Learning Opportunities ........... ees eeeeseseceecseeeeeeecaeeaeeseesaecseeseeeceesecneeeneeseesaeeatees 28 


Figure F.1: Teacher-Reported Average Usefulness Ratings of Preparation Experiences by 
COMPClONCY ALOE. sscsccsesiccscsccdiensecseessscscestvecseeedsccscuetceesveusssten sdecessuodcestschseeesseueessceensnssbanveassceetsh csgsetecsvensseesée 31 


Figure F.2: Means and 95 Percent Confidence Intervals for Teacher-Reported Usefulness Ratings 
and Ratings of the Frequency of Their Preparation Experiences With the 13 Competency Areas 
(in Order of Increasing Usefulness) ..........ccceccesscesscesssessecssecssecesecesecsseceseceeeeeseecseecsaecsaecaeceseceeseneeeaseeaseesaes 33 


Figure H.1: Teacher-Reported Frequency of Preparation Experiences With Competency Areas by 
CGPEICATION OULE rs sntuccaterrusencasoutdstnaniavvaccensstaddiadncceueeovenuadycanesuyanineladiaeledntedaumensmancensunreomeuises 39 


Figure H.2: Teacher-Reported Frequency of Preparation Experiences Through Types of Learning 
Opportunities by Certification ROUtC............cccccccsssessseeseesseceseceseceseceseceseceeeeessecseecsaecsaecaeceseceseeeueeesseesseeaaes 40 


Figure J.1: Variation in Preparation Experiences With Maintaining a Positive Classroom Climate 
Within Degree Programs for Three Preparation Providers .............cscsscseccesscsceeeeecesecseeeceeseceaeeaeeseceeeaeentees 50 


Figure K.1: Using Alternative Sets of Covariates in Models to Estimate Relationships of 
Preparation Experiences to Effectiveness Produced Estimates That Were Similar to Those From 
the: Primary Model yccisiiscccsiets scoest sseecdeceeideiis acdeacedeatuseceseeias sedi dayocduthasecdut beveesuaaebeazels seuaealiseededieseauelsseedavians 54 


Figure L.1: Relationships of Preparation Experiences With Creating a Productive Learning 
Environment or With Promoting Analytic Thinking Skills to ELA and Math Teaching 
Effectiveness for English Learners .................cccccsssscsseseccesecsoecossesescesossoesenceseeescosesseneeeoseseescecesoesenseneenetees 59 


NCEE 2019-4007 Teacher Preparation Experiences and Early Teaching Effectiveness ii 


List of Tables 


Table A.1: Teacher Survey Response Rates by District/State (N = 30) oo... eeeeeesecseeeeceseceeeeeeeeceeeeeaeeaeeeneeaee 1 
Table A.2: Distribution of Teachers by Year in Analysis Samples (Full Sample and Subsample).............. 2 
Table C.1: Constructs in Classroom Observation Instruments Used in the MET Project ............c:ccseeseeeee 7 


Table E.1: Measures of Teacher Preparation Experiences Derived From Exploratory Factor 
PMALYSOS vciveesssccszecedieeseledecevstanteivasseadcesss sicds'esseveesaceezeteascesteessecevansstcsdensccesanesteeestaarae¥ins savecdestatessasevsdens levees ees 28 


Table F.1: Average Measures of Usefulness 0.0.00... esesscsescssesseesceseesesecceseeneeeeceseeaeaecaeecesesseseaceneeseeeasees 29 


Table F.2: Correlations Between Teacher-Reported Usefulness Ratings and Ratings of the 
Frequency of Their Preparation Experiences With the 13 Competency Areas (in Order of 


Increasing Gorrel ation) seisessss¥scssesnvcsteegsinsvssivencesecavesnes sts sedsea ce ctuevsesnessaedada so desedp oubesbopntnddeasssnegnevtetatvensoteesaes 32 
Table G.1: Loading of 11 Competency Areas on Two Broad Categories ...........:ceecssecsceeceeeceeceeeeeeeeeeaeens 35 
Table G.2: Items and Loading Used to Create Two Broad Categories Within Each Type of 

T@arming OpPOreuDly.. .csosicecevcsseeves coreeacsseteusssesevesssevces sbeesadscoossueooseedeecsvenssusssesuussssendpsstessessossseesssesdesesedeesese 36 
Table H.1: Descriptive Statistics of Preparation Experiences With Competency Area and Type of 
Learning Opportunity by Certification Route 00.0... ccc eecesccsceeecesecsecseeeeeeseceseeaeesceeaecaeeaeeeeeeseceeeeaeeseesaeeatees 41 
Table I.1: Example Cells for Grade 5 Math Student ..0........sceesescscsesceseceeceseesceeecesecseeeceesecneeeeeeneeseesaeeaeees 44 


Table J.1: Decomposition of the Total Variance in the Average Frequency of Preparation 
Experiences With Competency Areas and Through Types of Learning Opportunities... eters 51 


Table L.1: Relationships of Preparation Experiences to ELA Teaching Effectiveness for 
Subgroups of Teachers Defined by Proportion FRPL-Eligible Students in Classrooms ............:.seeseeseeeee 56 


Table L.2. Relationships of Preparation Experiences to Math Teaching Effectiveness for 
Subgroups of Teachers Defined by Proportion FRPL-Eligible Students in Classrooms............::csseeseeseeeee 57 


Table L.3: Relationships of Preparation Experiences to Teaching Effectiveness in ELA and Math 
for Subgroups of Teachers Defined by Class Average Baseline ELA or Math Performance.................+++ 58 


Table L.4: Standardized Regression Coefficients for Relationships of Preparation Experiences 
With Effective Instruction for English Learners to ELA and Math Teaching Effectiveness for 
Hoplish: Learmers:.:.:cdscisssessaeksseiaehisevedh ss soebeehessksehsaviaddvsvasddusasadacisbevseeibeseteceassaedbianceetels scesstuasaedeteewsdevl. debeseeses 59 


Table M.1: Standardized Regression Coefficients for Relationships of Preparation Experiences 
With Creating a Productive Learning Environment to ELA Teaching Effectiveness by Year of 
H@AGHING va seivssxcesviscceseeavenetveeestteshacees es tenn beet cdcages cause bdanceleesveressevaeeeveea nea usb ncvevest vedi ancenvdssdeneasbceaeeeueesedeass 60 


Table M.2: Standardized Regression Coefficients for Relationships of Preparation Experiences 
With Creating a Productive Learning Environment to Math Teaching Effectiveness by Year of 
A CACHING s, Sei ei codavehancessecdnccavexssvaviisscaleakie cdew cases canssavisasecceuteasieslehaavetsbedensveel ashieesedacseuivesstsistes leashed vieedveetads 61 


Table M.3: Standardized Regression Coefficients for Relationships of Preparation Experiences 
With Promoting Analytic Thinking Skills to ELA Teaching Effectiveness by Year of Teaching .............. 61 


Table M.4: Standardized Regression Coefficients for Relationships of Preparation Experiences 
With Promoting Analytic Thinking Skills to Math Teaching Effectiveness by Year of Teaching.............. 61 


NCEE 2019-4007 Teacher Preparation Experiences and Early Teaching Effectiveness _ iii 


Appendix A. Teacher Survey Response Rates by District/State 


Table A.1: Teacher Survey Response Rates by District/State (n = 30) 


Teacher Responses Eligible Teachers 
Number of Number of Response Rate Number Eligibility 
District/State Teachers Notified Responses (%) Eligible Rate (%) 
1 77 67 87 30 45 
2 15 13 87 4 31 
3 364 304 84 252 83 
4 45 36 80 17 47 
5 245 179 i 31 17 
6 63 44 70 24 55 
7 265 183 69 74 40 
8 167 115 69 77 67 
9 1,480 942 64 281 30 
10 411 251 61 99 39 
11 164 97 59 50 52 
12 277 158 5 116 73 
13 2,421 1,371 57 552 40 
14 77 43 56 33 77 
15 596 327 55 184 56 
16 191 102 53 75 74 
17 552 291 53 208 ll 
18 129 66 51 37 56 
19 111 56 50 33 59 
20 69 34 49 28 82 
21 108 51 47 25 49 
22 343 145 42 52 36 
23 177 71 40 52 08 
24 196 73 37 61 84 
25 1,072 345 32 175 Bill 
26 92 28 30 26 93 
27 3,734 1,122 30 620 55 
28 98 17 17 16 94 
29 1,230 186 15 55 30 
30 296 12 4 7 58 
Overall 15,065 6,729 45 3,294 49 


NOTE: For the first 27 districts/states, the study team used data from the district/state to develop a list of potentially eligible teachers 
and fielded the survey to those teachers. The last three districts/states (27-30) did not want to share teacher contact information with 
the study team and so opted to send the survey link to teachers on the study's behalf. 
TABLE READS: In District/State 1, of the 77 teachers who were notified, 67 (87 percent) responded to the survey. Of the teachers 
who responded, 30 (45 percent) were eligible. 
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Table A.2: Distribution of Teachers by Year in Analysis Samples (Full Sample and Subsample) 


Full Sample 1,080 1,131 1,083 3,294 


Subsample 763 | 895 875 2,533 


TABLE READS: The distribution of teachers in the full sample was 1,080 first-year teachers, 1,131 second-year teachers, and 1,083 
third-year teachers. 
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Analyses and the Full Sample 


This appendix presents the results of analyses that compare the subsample of study teachers used for 
relational analyses (referred to hereafter as the relational sample) to the full sample of teachers who 
responded to the survey. Comparisons are presented on the geographic representation of states and 
preparation providers, the characteristics of districts, and the routes to certification and degree programs 
of teachers. 


Fewer states are represented in the relational sample than in the full sample, but 
their geographic dispersion is similar. 


The relational sample includes 14 states, four fewer than the full sample. The four states not in the 
relational sample but in the full sample are Alabama, Minnesota, Nevada, and Wisconsin. As in the full 
sample, the states that teachers taught in were spread across the country but the majority of them were in 
the southern and western regions of the United States. In both samples, the highest numbers of study 
teachers are (in descending order) in New York, Florida, Oklahoma, Georgia, Texas, and Illinois. 


Both the relational sample and the full sample included a large number of 
preparation providers in almost all states. 


Study teachers in the relational sample were prepared by 469 providers in 44 states (all states except 
Hawaii, Montana, North Dakota, Vermont, Washington, and Wyoming) and the District of Columbia and 
Puerto Rico, compared to 566 providers in 46 states (all states except Hawaii, Montana, Vermont, and 
Wyoming) and the District of Columbia and Puerto Rico in the full sample. The two states represented in 
the full sample but not in the relational sample are North Dakota and Washington. 


The characteristics of districts in which teacher taught were similar in the 
relational and the full sample. 


Figure B.1 compares the characteristics of districts in the relational sample (n = 169) and those in the full 
sample (n = 242). District characteristics are weighted by the number of teachers in each district. 


More than half of teachers in both the relational and full samples teach in districts that are in the South 
(63 and 60 percent, respectively) and nearly all serve more than 2,350 students (93 and 95 percent, 
respectively). Fifty-six and 58 percent of the districts in in which teachers taught in the relational sample 
and the full sample, respectively, are urban. For the relational sample, over four-fifths of teachers teach in 
districts where at least 50 percent of students who are eligible for free or reduced-price lunch (FRPL), and 
two-thirds of teachers teach in districts with 33 to 67 percent minority students. Those same 
characteristics for the full sample are almost identical. In both samples, about half of teachers teach in 
districts that have greater than 10 percent English learners.! 


Although some of these differences are statistically significant, it should be noted that when testing the 
statistical significance of differences between two large samples, small differences—such as 1 or 2 percentage 
points—can be statistically significant. The large sample sizes and the small size of the difference lead us to 
conclude that the characteristics of the districts in which teachers taught are similar for the two samples. 
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Figure B.1: Percentages of Study Districts by Demographic Characteristics in the Relational and Full Samples, Weighted by Number of 
Teachers 
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FRPL is free or reduced-price lunch eligibility, a proxy for low-income status. 

NOTE: Characteristics are weighted by the number of study teachers in each district. District size classifications are based on quartiles of the national distribution of districts in the 
Common Core of Data as follows: “Large” > 2,350 students; “Mid-Size” 800—2,350 students; “Small” 300-799 students; “Very Small” < 300 students. Minority indicates any of the 
following races based on the CCD: American Indian/Alaska Native, Asian, Hispanic, Black, Two or More Races. Sample size differs across district characteristics in the full sample due 
to missing data in the CCD. From the full sample of 3,294 teachers in 242 districts, sample size varied from 3,290 to 3,294 teachers in 240 to 242 districts. Sample size is 2,533 in 169 
districts in the relational sample. Chi-square tests for the equality of the distribution of census region, urbanicity, size, and percent minority rejected at the 5-percent significance level. 
Chi-square tests for the equality of the distribution of percent FRPL and percent English learners for the full and relational samples failed to reject at the 5-percent significance level. 
FIGURE READS: In the relational sample, 60 percent of study districts were in the South census region, 7 percent were in the West, 10 percent were in the Midwest, and 23 percent 
were in the Northeast. In the full sample, 63 percent of study districts were in the South census region, 9 percent were in the West, 10 percent were in the Midwest, and 19 percent 
were in the Northeast. The distribution of census regions was statistically significantly different for the relational and full samples. 

SOURCE: U.S. Census for region. CCD for 2013-14 for urbanicity, size, percentages FRPL, minority, and English learners. 
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The percentages of teachers entering teaching through traditional and 
alternative routes to certification were similar for teachers in the relational and 
full samples. 


Eighty (80) percent of teachers in the relational sample entered teaching through a traditional route to 
certification versus 78 percent in the full sample (figure B.2).? 


Figure B.2: Percentage of Study Teachers by Certification Route in the Relational and Full 
Samples 
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NOTE: Sample size is 2,530 in the relational sample and 3,291 in the full sample. Route to certification information was missing for 
three study teachers as a result of survey nonresponse. A chi-square test for the equality of the distribution of routes to certification 
for the relational and full samples was rejected at the 5-percent significance level. 

FIGURE READS: Eighty percent of teachers in the relational sample entered teaching through traditional routes to certification and 
20 percent entered teaching through alternative routes to certification. 

SOURCE: Study’s teacher survey data, 2015. 


The percentages of teachers entering teaching through different types of degree 
programs were similar for teachers in the relational and full samples. 


Fifty-one (51) percent of study teachers in the relational sample entered teaching through a traditional 
undergraduate program culminating in a four-year bachelor’s degree, whereas the percentage was 52 in 
the full sample (figure B.3). Three (3) percent of teachers in the relational sample and 2 percent in the full 
sample entered teaching through a traditional post-baccalaureate or fifth-year program. Twenty-six (26) 
percent of teachers in the relational sample versus 23 percent in the full sample entered teaching through a 
traditional master’s degree or a joint bachelor’s/master’s degree program. The remaining study teachers in 
the relational sample indicated that they entered teaching though an alternative route to certification, with 


Although this difference is statistically significant, it should be noted that when testing the statistical 
significance of differences between two large samples, small differences—such as 1 or 2 percentage points— 
can be statistically significant. The large sample sizes and the small size of the difference lead us to conclude 
that the routes to certification by which teachers entered teaching are similar for the two samples. 


NCEE 2019-4007 Teacher Preparation Experiences and Early Teaching Effectiveness 5 


8 percent of the relational sample receiving a master’s degree and 12 percent not receiving a master’s 
degree. The percentages in the full sample were 8 and 13, respectively.* 


Figure B.3: Percentage of Study Teachers by Degree Program in the Relational and Full Samples 
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* indicates a t-test rejects the null hypothesis of equivalence between the mean for teachers in the relational and full samples at the 
5-percent significance level. 

NOTE: Sample size is 2,530 in the relational sample and 3,291 in the full sample. Degree program information was missing for three 
study teachers as a result of survey nonresponse. Totals do not add to 100 percent due to rounding. Totals in this figure also do not 
match those in figure B.2 due to rounding. 

FIGURE READS: Fifty-one percent of teachers in the relational sample and 52 percent in the full sample entered teaching through 
traditional routes to certification and obtained a bachelor’s degree. 

SOURCE: Study’s teacher survey data, 2015. 


3 Although some of the differences described above were statistically significant, it should be noted that when 
testing the statistical significance of differences between two large samples, small differences—such as 1 or 2 
percentage points—can be statistically significant. The large sample sizes and the small sizes of the differences 
lead us to conclude that the degree programs by which teachers entered teaching are similar for the two 
samples. 
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This appendix includes figures that are relevant to the development of the teacher survey and references 
in chapter 2 as well as a description of the cognitive testing that we conducted to ensure that the survey 
items would be understandable to teachers and that the survey format would be easy to navigate. 


Table C.1 lists the constructs from the classroom observation instruments that were used in the Measures 
of Effective Teaching (MET) Project and on which we based the instructional strategies for each 
competency area in the survey. 


Table C.1: Constructs in Classroom Observation Instruments Used in the MET Project 


Framework for Teaching 
(FFT) 
(Scale: 1 = Unsatisfactory, 2 = Basic, 
3 = Proficient, 4 = Distinguished) 


Classroom Assessment 
Scoring System, K-3 
(CLASS) 
(Scale: 1/2 = Low, 3/4/5 = Medium, 6/7 = High) 


Domain 2 = Classroom Environment 
e Creating an environment of respect and rapport 
e Managing student behavior 
e Managing classroom procedures 
e Organizing physical space 
Domain 3 = Instruction 
e Establishing a culture for learning 
e Using assessment in instruction 
e Demonstrating flexibility and responsiveness 
e Engaging students in learning 
e Using questions and discussion techniques 
e Communicating with students 


Domain 1 = Emotional Support 
e Positive climate 
e Teacher sensitivity 
e Regard for student perspectives 
Domain 2 = Classroom Organization 
e Behavior management 
e Productivity 
e Negative climate 
Domain 3 = Instructional Support 
e Instructional learning formats 
e Analysis and inquiry 
e Quality of feedback 
e Instructional dialogue 
e Content understanding 


Protocol for Language Arts Teaching Observation-Prime 
(PLATO Prime) 
(Scale: No evidence, Limited evidence, Evidence with 
some weaknesses, Consistently strong evidence) 


Mathematical Quality 
Instruction Lite 
(MQI Lite) 
(Scale: Low, Medium, High) 


e Behavior management 

e Time management 

e Intellectual challenge 

e Classroom discourse 

e Strategy use and instruction 
e Modeling 


Classroom work connected to math 

Absence of errors and imprecision 

Explicitness and thoroughness 

Working with students on mathematics 

Richness of math content 

Student participation in meaning-making and reasoning 


Lesson-based guess at mathematical knowledge for 
teaching 
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Figure C.1 shows the steps that we used to construct the survey questions with competency areas and 
types of learning opportunities. 


Figure C.1: Steps in Constructing the Survey Questions on Teacher Preparation Experiences 


TEACHER SURVEY DEVELOPMENT 


COMPETENCY AREAS TYPES OF LEARNING OPPORTUNITIES 


Specified metric for rating level of ) 

experiences with each teaching strategy 

(“Never/Rarely” to “Very often”) | 
. . 5 


PREPARATION EXPERIENCES 


a 


TEACHERS’ EXPERIENCES LEARNING ABOUT COMPETENCY AREAS 
THROUGH DIFFERENT TYPES OF LEARNING OPPORTUNITIES 
(COURSEWORK, OBSERVATION, PRACTICE, FEEDBACK) 


Cognitive Testing 


We did two rounds of cognitive testing. The first round focused on teachers’ understanding of the 
wording and content of the items in the grids. This round of cognitive testing was also used to obtain 
teacher feedback on the five-point frequency scale and whether they found it challenging to use a scale 
that only had labels at the endpoints. The second round focused on teachers’ ability to navigate through 
the survey sections and to complete the multiple items within the survey grids. 


Round 1 


We conducted cognitive testing of the survey with six teachers in grades 4—6 who were in their first three 
years of teaching. We instructed the teachers to complete paper versions of the survey items on the 
competency areas (one survey grid for each) during a phone interview. Each survey question required the 
teacher to provide a frequency rating for each of five instructional strategies for each of four types of 
learning opportunities. After the teacher completed a survey question, we discussed the teacher’s 
understanding of the meaning of the instructional strategies and of the distinctions among the learning 
opportunities, using a set of common probes. We also asked teachers about their perceptions of the five- 
point frequency scale, including whether they were comfortable selecting a point of the scale that 
described the frequency of their experiences and if they could distinguish different frequency levels 
across strategies and types of learning opportunities. Figure C.2 below provides the probes used about 
respondent understanding of the descriptions of the strategies in the grid for productive use of classroom 
time. 


Figure C.2: Round 1 Cognitive Testing Probes for Productive Use of Classroom Time 


The first instructional strategy is: Providing clear options/directions for students if they finish a task or in- 
class assignment early (e.g., students know which work centers to go to after they finish their work). 
¢ What does this look like to you in the classroom? 


¢ What teaching or instructional behaviors do you think about when you read this strategy? (Record 
teacher’s description in the interviewer reporting form) 


If the teacher describes student, rather than teacher behaviors, probe: 


¢ What would teacher behaviors look like? What might the teacher do? How might the teacher act? 


If the teacher’s description matches that of the developer, move on to the next strategy. 
If the teacher’s description is different from that of the developer, probe: 


The developers of this strategy described it this way (read developer definition from interviewer 
reporting form). 


What words in the strategy led you to think of the behaviors as you did? 


What words might we have used to help you think about the strategy in a more similar way to what 
the developer intended? (Record suggested language in interviewer reporting form) 


Move on to the next strategy. 


NCEE 2019-4007 Teacher Preparation Experiences and Early Teaching Effectiveness 9 


The results of this round of cognitive testing showed that teachers in general recognized and felt that they 
understood the descriptions of the instructional strategies and were comfortable rating the frequency of 
preparation experiences that they received on these strategies separately for the different types of learning 
opportunities. Teachers did not report being confused or distracted by the vocabulary or concepts in the 
items. The fact that we pulled these items from commonly used instruments that teachers are familiar with 
to rate/describe instruction increased our confidence that teachers would find the language acceptable and 
understandable. We did make minor revisions to wording based on teachers’ recommendations. 


Round 2 


We conducted a second round of testing using of the online version of the survey. This round of testing 
focused on teachers’ understanding of how to complete the survey, using the instructions embedded in the 
online version. We were interested in whether the features of the online version, which we had introduced 
to reduce confusion and burden of the grid questions, did in fact make completing the survey reasonable 
for teachers. These features included visual cues, such as color-coding each grid as well as rows within 
grids; drop-down menus for each cell so that teachers could easily select their ratings; different options 
for moving through the items—tabs or arrow keys; and having the shading of a cell change color when it 
was completed to help teachers know which of the multiple cells in each grid had been answered. 


We tested the online survey with five teachers in grades 4-6 who were in their first three years of 
teaching. Each of the five teachers completed the survey within 30 - 45 minutes. All teachers found it 
easy to navigate through the survey as well as within the grids, and found the grid instructions clear. The 
teachers made suggestions for reducing burden and adding encouragement within the grids, such as 
adding a meter that indicated the percentage of survey completion, highlighting certain sections of the 
grids, and adding hotlinks back to the instructions. We made these revisions before launching the survey 
to study teachers. 
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Study of Teacher Preparation Experiences and Early Teaching Effectiveness 
Teacher Survey 


Section 2: Teacher Preparation Experiences 


The next questions ask about your experiences while attending your teacher preparation program for your initial certification. Please 
consider all of your preparation experiences in the program you attended for initial certification, but do not include experiences after 
you received your initial certification such as experiences in a master’s program or a program to obtain additional certifications. 
These questions are the heart of the survey. This is the first time teachers are being asked about their preparation experiences in 
this way. We hope you find it interesting and value your responses to each question! 


Instructions: There are 13 questions in this section covering 13 competency areas. The rows on each question list teaching strategies specific to the 
competency area. The columns describe four types of learning opportunities. Use the drop-down menu to select a response category 
that best describes, for each instructional strategy, how often as part of your preparation program for your initial certification you 
had experience with the instructional strategy through each of the four types of learning opportunities. In the fifth column, for each 
instructional strategy, select the response category that describes how useful your preparation experiences have been for your 
classroom instruction. 
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Section 2: Teacher Preparation Experiences 


(Question 1 of 13: Facilitating Extended Classroom Discussions) 


As part of your preparation program for initial certification, how often did you: 


Observe a teacher 
using this strategy in 


a K-12 classroom 


Receive feedback 
on your use of this 
strategy from 
program staff ora 
cooperating teacher 


Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 

(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 

~~ y y — S 3 
o o o o a = ne 
g 5| 8 5| 8 5| 8 s|s/ 3/2] |3 
S, SS S| 8 iS a =| 4) 3] 5/3 

Teaching Strategies for “Facilitating Extended S = S = S = c = S ~ S ro - 
Classroom Discussions” ag >| & >| & >] & =>] =]; =| 4} 5/ > 
Holding extended classroom discussions about lesson 
content where students are asked to explain and defend rl2leltalsliioeleslals 1/2le3tlals 
their thinking, compare their ideas to others, brainstorm 
about new ideas, and formulate hypotheses. 
Asking students to explain and clarify their ideas about 
academic content by using methods such as open-ended | 1/2/])3;,4/5{]1j;]2{|3/4)]5 iE | 2) eh | eS 
questions, repetition, or extension. 
Using questioning and discussion techniques to challenge 
students to build logical arguments, challenge their 1/2;)3)4;5;)1/2)3)4)]5 1/2/;3)4)]5 
remises, and critique the arguments of others. 
Modeling “active listening” during classroom discussions, 
i.e., using non-verbal cues such as making eye contact, ree (ee eae (| el eee ee lemme! 1) 2) 3 | 4 | & 
leaning forward, nodding. 
Using methods such as open-ended questioning, 
repetition and extension to encourage students to use 1J/o2laltalslii2elesltalslii2lelalsl1)2elatalslaleolealals 
complex language to explain and clarify their ideas about 
academic content. 
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Section 2: Teacher Preparation Experiences 


(Question 2 of 13: Maintaining a Positive Classroom Climate) 


As part of your preparation program for initial certification, how often did you: 


Observe a teacher 
using this strategy in 
a K-12 classroom 


Receive feedback 
on your use of this 
strategy from 
program staff or a 
cooperating teacher 


Communicating respect and warmth both in words and 


Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 

(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
2 
a B 

_ _ _ oy 
a) © a) © o 3 = 
g 5| 8 5| 8 5| 8 a =| 3\2 3 

fe) 
= 6) 5) S S| = a =| 4) #| 5/3 
Teaching Strategies for “Maintaining a Positive S = S - S = S Sp! e|s| § o 
Classroom Climate” a = || @e >| & >| & S| 22 1 2iL@ 25 


working together and sharing materials. 


Responding to students’ efforts and participation with 
positive comments (e.g., “What a great idea!,” “You guys 
are working together really well on that project.”). 


non-verbally (making eye contact, being in physical 1 511 slilol3tlalsl21i2la3lalslalelelals 
proximity) when interacting with individual students and 

the class as a whole. 

Demonstrating knowledge of and interest in students’ 

lives inside and outside of school. : eal PlAle lol lol lel dL ow a EE 
Listening closely and with genuine interest when students 1 511 slil2l3ialsl21i2l3lalslaleolelals 
talk and encouraging students to listen to each other. 

Encouraging cooperation among students, such as 1 514 oer Cmeemmieem| 7) >/3/]/4/5 
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Section 2: Teacher Preparation Experiences 


(Question 3 of 13: Productive Use of Classroom Time) 


As part of your preparation program for initial certification, how often did you: 


Observe a teacher 
using this strategy in 
a K-12 classroom 


Receive feedback 
on your use of this 
strategy from 
program staff or a 
cooperating teacher 


Providing clear options/directions for students who finish 
class activities early (e.g., students have other activities 1/2;);3;),4;5;)1/2)3)4)5 
to do when they finish their work). 


Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 

(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
= 
a B 

— _ Ny 5) 
© © a) © o 3 = 
8 5| 8 5| 8 5| 8 a =| 3\2 3 

fe) 
S eS SES SS oy =| 4) z| s/ 3 
Teaching Strategies for “Productive Use of S = S S eS S Al a| si § o i 
Classroom Time” a = || @e >| & >| & = | se 2 ee Ss 


address student behavior. 


Providing clear guidance to students about what is 
expected during transitions between activities and 
different groupings to avoid losing instructional time to SS SS ee | coe ie yee 


Establishing procedures for students during whole-class 
discussions or activities to avoid losing instructional time 
to address student behavior (e.g., raising hands to talk in 
turn, listening actively to other students). 


Conveying what students are expected to do during a 
lesson when they are working independently or with 
peers without direct teacher supervision. 


Teaching students to implement classroom routines such 
as distribution and collection of materials with minimum 
disruption to the flow of instruction. 
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Section 2: Teacher Preparation Experiences 
(Question 4 of 13: Demonstrating Sensitivity to Student Needs) 


As part of your preparation program for initial certification, how often did you: 


Observe a teacher 
using this strategy in 
a K-12 classroom 
(include videos and 
direct observations 


Read about, hear 
about or see a role 


Receive feedback 
on your use of this 
strategy from 
program staff or a 
cooperating teacher 
that included 
information about 


Practice this 
strategy in a K-12 


How useful have 
your preparation 


play of this strategy | during your fieldwork classroom prior to what you did experiences been 
(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
= 
_ — _ y zg B 
o o o o a ES. 
g s| 5| 3 s| 3 s|s|sl2] |3 
S 8| S$ 8| $ 8| 8] §/ 8) 8] a] s 
Teaching Strategies for “Demonstrating Sensitivity S = S = S i g ss 5s) § o = 
to Student Needs” oa Sms >| & >| & = | a8 2 | @ Sis 
Recognizing the signs that students may need extra 1 511 als5il1lola3talslaii2lel)alslailols3ials 
support or assistance. 
Checking in with students who are having difficulties with 
classroom activities or who are not engaged in the 1 2a ee ee ee | ele On em com! 1) 2 | 3 | 4 | & 
activities. 
Providing individualized support to both students 
struggling with understanding content and students who 17/;/2;);3)4;5;)1/2)3;)4)/5712/2)3)4;)5;)712)2)3/4)/5 
need extra challenge. 
Adjusting pacing and wait time during instruction if 
students need more time to understand a concept, 1 2a one een ome lee een ee On en em eomimeom| 1) 2) 3 | 4) 5 
process information, or complete an assignment. 
Follow-up with students who have experienced difficulty 
to ensure that the additional assistance offered has met 1)/2;);3);4/;57;12}),2/3/4)5;7;1);)2)3;);4;/5;2)2),3/4)5 
their needs. 
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Section 2: Teacher Preparation Experiences 


(Question 5 of 13: Conveying the Importance of Learning) 


As part of your preparation program for initial certification, how often did you: 
Receive feedback 
on your use of this 


Observe a teacher strategy from 
using this strategy in program staff or a 
a K-12 classroom cooperating teacher 
Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 
(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
= 
a) 
a) © a) © a 3 = 
g 5| 8 5| 8 5| 3 s}s/s| 2] | 3 
S 3| 5 °| § 8| $ S| £| 8] 3| s| 3 
Teaching Strategies for “Conveying the Importance S = S = S = S Al a|s| § o = 
of Learning” ao >| & => || @e >| & > |L se |_| @ Ss 
Conveying to students the importance and value of the 1)/2)/3lalsliloleatalsili1i2lelalsliai2ele)alslaileolealals 


concepts and skills they are learning. 
Conveying to students that their job in the classroom is to 

work hard and engage in learning. 1 |e Soe ete |e ee lee On |e memo! 1) 2) 3 | 4] 5 
Conveying high expectations for student participation and r1Jo2latalsl2aie2leltalslii2lelalsi1)2latalslal2lealals 
engagement in classroom tasks and activities. 
Conveying the belief that all students are capable of 1lelelalseleleleale-eials «lelelalelse\-l. RRR 
learning challenging material if they work hard. 
Conveying that students should take responsibility for the 


quality of their own work by initiating improvements, 1/2lalalsl1il2lelalslii2}ealalst1)2olatalslaleoelalals 
making revisions, or adding detail based on feedback 


from the teacher. 
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Section 2: Teacher Preparation Experiences 


(Question 6 of 13: Managing Student Behavior to Maximize Learning Time) 


As part of your preparation program for initial certification, how often did you: 


Observe a teacher 
using this strategy in 
a K-12 classroom 
(include videos and 
direct observations 


Read about, hear 
about or see a role 


Practice this 


strategy in a K-12 


Receive feedback 
on your use of this 
strategy from 
program staff or a 
cooperating teacher 
that included 
information about 


How useful have 
your preparation 


play of this strategy | during your fieldwork classroom prior to what you did experiences been 
(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
2 
a B 
y y x oy 
a) © © © wH S| = 
g 5| 8 5| 3 5| 3 s}s/s|2] | 3 
ie) 
S 3| 5 °| § 8| $ S| =| 8] 3| s| 3 
Teaching Strategies for “Managing Student Behavior | © = S - S eS S Al a| si § > ce 
to Maximize Learning Time” oa Sms >| & >| & = | sei 2 ae Ss 


Providing consequences for misbehavior that are 
enforced in a consistent and predictable way, whenandif | 1}2/3)/4/5]1/2/3])4]5],1]21|3 


necessary. 
Redirecting misbehavior using nonverbal cues such as 

eye contact, gestures, or physical proximity to students. Be ee ee eee © | 2 | 2 | 4 | CO 
Monitoring students to anticipate and redirect problem rlolealtalsliai2leltalsliie2lelalsi1)2elatalslaleolalals 
behavior to minimize loss of instructional time. 

Encouraging students to monitor themselves, i.e., to be son ce een ee eet! 7 | >/ 3/4 / 5 


responsible for their own behavior. 

Providing students with expectations for behavior that are 
clear, consistent and understood by everyone in the 1/2;);3)4;5;)1/2)3)4)5)1442 4) 3 
class. 
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Section 2: Teacher Preparation Experiences 


(Question 7 of 13: Building Comprehension of Academic Concepts) 


As part of your preparation program for initial certification, how often did you: 


Observe a teacher 


using this strategy in 
a K-12 classroom 


Receive feedback 
on your use of this 
strategy from 
program staff or a 
cooperating teacher 


Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 

(such as during becoming a full-time well/how you could for your classroom 
coursework)? teacher? improve? instruction? 
2 
a B 

— _ Ny 5) 
© © © © o 3 = 
8 5| 8 5| 8 5| 8 a =| 3\2 3 

Ie) 
S eS SES SS a =| 4| z| s/ 3 
Teaching Strategies for “Building Comprehension of | £ = S = S eS S el a|s| § oO re 
ug >| x >| « >] & Sy) 2) oS) = 


Academic Concepts” 


Helping students understand and explain multiple 
perspectives about academic concepts, such as different 
views on immigration or global warming, and to support 
their position. 


Anticipating areas where student misconceptions about 
content are likely and providing explanations that address 
the misconceptions. 


Modeling the use of precise academic vocabulary and 
explaining its meaning. 


Providing opportunities for students to practice using new 
strategies and processes in context and monitoring their 
understanding and appropriate use. 


Requiring students to analyze and explain their ideas or 
answers about what they have read and written, for 
example, asking students to summarize and defend a 
position they have taken, using citations or other 
evidence. 
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Section 2: Teacher Preparation Experiences 
(Question 8 of 13: Providing Feedback that Helps Student Learning) 


As part of your preparation program for initial certification, how often did you: 


Observe a teacher 
using this strategy in 
a K-12 classroom 


Receive feedback 
on your use of this 
strategy from 
program staff or a 
cooperating teacher 


Using questions, prompts and assessments to regularly 
monitor student understanding and diagnose 
misconceptions or errors in thinking. 


Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 

(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
= 
a) 

o o o o a 5 = 
g 5| 2 5| 8 5| 3 S}s/S|=/ | 3 
S 8| = 8| = S| 8} 5/8] sf a] 3 

Teaching Strategies for “Providing Feedback that S >| & >| & a) & S| S| 2] €/ a] & 
noe te G| & G| & G| & G| ©} S| S|] H| G& 
Helps Student Learning & >| & >| & >| & =] =]; 2/49} 5/ > 


Providing students with strategies and opportunities for 
monitoring their own understanding of academic concepts 
and learning progress. 


Teaching students how to provide constructive review 
and criticism of each other’s work. 


Providing assistance, hints, or prompts that help students 
deepen their understanding of concepts or procedures 
and perform at a higher level. 


Providing feedback that builds on learning by expanding 
on and clarifying student ideas, providing new 
information, and correcting any misstatements or 
misconceptions. 
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Section 2: Teacher Preparation Experiences 
(Question 9 of 13: Building Students’ Higher-Order Thinking Skills) 


As part of your preparation program for initial certification, how often did you: 
Receive feedback 
on your use of this 


Observe a teacher strategy from 
using this strategy in program staff or a 
a K-12 classroom cooperating teacher 
Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 
(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
= 
— — — — S 3 
a) a) © a) o 3 = 
8 5| 8 5| 8 5| 8 >: aes 3 
S eS SHES SS =| 4) 3] 5) 3 
Instructional Strategies for “Building Students’ S = S = S = S a! =| s5/ § > = 
Higher-Order Thinking Skills” o& >| & >| & >| & = | sei 2 | ae Ss 
Presenting students with open-ended problems and tasks 1|/2}l3lalslilolatalsili1i2elaslalsliali2ele)alsliaiolealals 


where there is not one correct answer or approach. 
Modeling the problem-solving process for students by 
“thinking out loud” to bring students’ attentiontothe steps | 1/2);3]4/5]1/)2;)3;/4)5;/;1/;2);3)4/,5;1)2)3;,4)57;1)2/3)44]5 
they are expected to go through to derive a solution. 
Asking students to explain or show their thinking—how 
they approached a given problem or task and how they 
arrived at their answer (e.g., doing a math problem at the 
board, arguing an interpretation of a poem or story). 
Providing assignments that engage students in content 
that is cognitively challenging, i.e., that requires them to 1 eZ ee et || eel fee ee ee ee on ee mcmieom! 1) 2 | 3 | 4 | 5 
make predictions, formulate hypotheses, or brainstorm. 
Providing materials and resources that promote deeper 
learning, such as primary source materials in social r1l2latalsl2aie2leltalslii2lelalsi1)2latalslaleolalals 
studies, or articles or published documents in language 
arts. 
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Section 2: Teacher Preparation Experiences 


(Question 10 of 13: Designing and Using Assessments of Student Learning) 


As part of your preparation program for initial certification, how often did you: 
Receive feedback 
on your use of this 


Observe a teacher strategy from 
using this strategy in program staff or a 
a K-12 classroom cooperating teacher 
Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 
(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
2 
— — y — e B 
© © © a) a 3 = 
g 5| 8 5| 8 5| 8 a s|s\2| |8 
= 6S eS eS =| 2) s| s| 3 
Teaching Strategies for “Designing and Using S = S - S = S al a|s| § o 
Assessments of Student Learning” oa Ss >| & >| & = | se | 2 | Ss 
Clearly defining assessment criteria for students. 1 5 | 1 5 | 1 5 | 1 o{[1/]2/3|4|{5 
Modifying assessments to meet the needs of individual feel liae B lia | 2 \-3 EB eietelleon les 5112 s5/1l2l3lals5 
students. 
Using results of assessments to guide planning of future 1\/2)3lalslailolatalsli1i2laslalslale2ele)alsliaiolealals 
instruction. 
Using questions, prompts and assessments to regularly He ee ee eee eee ee el ee eer ee meomemem! 4 | >| 3 | 4/5 
diagnose student learning. 
Providing students with opportunities to monitor their own 1)/2l3alalslilolatlalsi1i2lalalslale2ele)/alslialolealals 
understanding and progress. 


NCEE 2019-4007 Teacher Preparation Experiences and Early Teaching Effectiveness 22 


Section 2: Teacher Preparation Experiences 
(Question 11 of 13: Effective English Language Arts Instruction) 


Please answer the questions about your preparation experiences in English language arts, even if you are not currently teaching this subject area. 


As part of your 


reparation program for initial certification, how often did you: 


Observe a teacher 
using this strategy in 
a K-12 classroom 


Receive feedback 
on your use of this 
strategy from 
program staff ora 
cooperating teacher 


Arts Instruction” 


Teaching students strategies to help in reading 
comprehension, such as the use of graphic or semantic 
organizers, and monitoring students for correct 
application and use of these strategies. 


Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 

(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
= 
© 

© o © o o S a 
g 5| 8 5| 8 5| 3 as) =|2| | 3 
S S| S 8) & |S ey =| 2) s/5\| 3 

Teaching Strategies for “Effective English Language | © S = S = S =) =| 35] § 9 = 
ag >| & >| «x >] & > | Pes ee Ss 


Visibly enacting strategies, skills, and processes, for 
example, leading a “think aloud” to show students how to 
identify the theme of a story or how to support a 
statement with a citation from text. 


Teaching students meta-cognitive strategies for 
monitoring their reading comprehension, such as pausing 
to ask themselves if they understood what they just read 
or what the author wanted them to know, and monitoring 
students for correct application and use of these 
strategies. 


Providing students opportunities for guided reading during 
which they can apply newly-learned comprehension 
strategies and receive performance-based feedback on 
their performance. 
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As part of your 


reparation program for initial certification, how often did you: 


Read about, hear 
about or see a role 


Observe a teacher 
using this strategy in 
a K-12 classroom 
(include videos and 
direct observations 


Practice this 
strategy in a K-12 


Receive feedback 
on your use of this 
strategy from 
program staff or a 
cooperating teacher 
that included 
information about 


How useful have 
your preparation 


Providing language arts assignments that require 
students to analyze, infer, explain their ideas, or justify 
their answers, for example, assignments involving text 
analysis, summarizing and evaluating information for an 
argument, or taking a position and defending it. 


play of this strategy | during your fieldwork classroom prior to what you did experiences been 
(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
2 
a B 
y y — oy 
a) © a) © aH 3 _ 
g 5| 2 5| 3 5| 3 s}s/s| 2] | 3 
fe) 
S 3| 5 °| § 8| $ 8| 5] 8] 3) 3) 2 
Teaching Strategies for “Effective English Language | © = S S S S al s|s| § ‘9 eS 
Arts Instruction” a = || @e >| & >| & = |L se | 2 | @& Ss 
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Section 2: Teacher Preparation Experiences 


(Question 12 of 13: Effective Mathematics Instruction) 


Please answer the questions about your preparation experiences in mathematics, even if you are not currently teaching this subject area. 


As part of your 


reparation program for initial certification, how often did you: 


Observe a teacher 
using this strategy in 
a K-12 classroom 


Receive feedback 
on your use of this 
strategy from 
program staff ora 
cooperating teacher 


Instruction” 


Helping students to use different mathematical 
approaches to solving a problem and explaining the 
advantages (efficiency, ease of use) and disadvantages 
of each. 


Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 

(such as during or student becoming a full-time well/how you could for your classroom 
coursework)? teaching)? teacher? improve? instruction? 
2 
a B 

y y — 5) 

a) © a) a) a 3 ee 
8 5| 8 5| 8 5| 8 a =| s\2 3 

Is) 
S oS SES S| & a =| 3) | s| 3 
Teaching Strategies for “Effective Mathematics S = S = S = S =! eis § ‘> = 
eg >| «x >| «x >] & = |Past 2 | aL SLs 


Asking students to explain why their solution to a math 
problem works and how they know their answer is right. 


Giving students problems to help them understand their 
thinking errors and what other methods might have 
worked better. 


Identifying the source of student errors or misconceptions 
and helping students understand why their method of 
solving a math problem didn’t work. 


Helping students understand how they use math in their 
lives, e.g., when they have to figure out how to double a 
recipe or to calculate the probability of a team winning by 
more than 2 points. 
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Section 2: Teacher Preparation Experiences 


(Question 13 of 13: Instruction for English Learners) 


As part of your preparation program for initial certification, how often did you: 


Observe a teacher 


using this strategy in 
a K-12 classroom 


Receive feedback 
on your use of this 
strategy from 
program staff or a 
cooperating teacher 


Read about, hear (include videos and Practice this that included How useful have 
about or see a role direct observations strategy in a K-12 information about your preparation 
play of this strategy | during your fieldwork classroom prior to what you did experiences been 

(such as during becoming a full-time well/how you could for your classroom 
coursework)? teacher? improve? instruction? 
2 
a B 

— = — y ) 
© © a) a) o 3 = 
8 5| 8 5| 8 5| 8 a =| 3\2 3 

fe) 
S eS SES Ss oy =| 4) z| 5/3 
Teaching Strategies for “Effective Instruction for S = S = S = S 1 eis § o re 
ug >| «x >| «x >] & > Re Rares 


English Learners” 


Teaching ELs word-learning strategies such as using 
cognates (words that share a common origin in multiple 
languages) or context clues, focusing on word parts, or 
practicing using different forms of a root word. 


Facilitating discussions in which students have assigned 
roles to increase opportunities for ELs to talk about 
academic words. 


Using strategies that help ELs process content and 
practice language, such as having students read and 
discuss short text passages, act out word meanings, or 
ask each other questions. 


Using modeling, hands-on materials, visuals, 
demonstrations, gestures, and videos to engage ELs and 
to stimulate group discussion. 


Creating small groups of students with varying levels of 
English proficiency so that students with stronger English 
skills can provide language models for less proficient 
students. 
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Appendix E. Measures of Preparation Experiences 


As depicted in appendix D, the survey asked novice teachers about experiences they obtained as part of 
their preparation program for initial teacher certification. Within each of the 13 competency areas, 
teachers described the frequency of their preparation experiences with five instructional strategies by 
assigning a rating from 1 to 5, where higher scores imply more frequent experiences. Instead of asking 
teachers to rate the overall frequency of their experiences related to a strategy, the survey asked teachers 
about the frequency of experiences with each strategy through four types of learning opportunities: 

(1) Coursework: reading or hearing about, or seeing a role play of the strategy (such as during 
coursework); (2) Observation: observing a teacher using the strategy in a K-12 classroom; (3) Practice: 
practicing the strategy themselves in a K-12 classroom; or (4) Feedback: receiving feedback from a 
supervisor/mentor teacher on their use of the strategy. 


This approach yielded 260 separate ratings from each teacher, because teachers rated the frequency of 
their preparation experiences related to 65 strategies for each of the four types of learning opportunities. 
Because this is too much data to report directly, we created three types of average frequencies for analysis 
and reporting. 


1. First, we computed an average frequency of preparation experiences obtained about each of the 
competency areas through each of the four types of learning opportunities. These scores 
represent the average frequency of preparation experiences obtained across the five instructional 
strategies within each competency area for each type of learning opportunity separately. For 
example, figure E.1 shows an average frequency of preparation experiences obtained with the 
competency area across the five strategies through Coursework of 3.6, through Observation of 
3.0, through Practice of 2.0, and through Feedback of 1.6. Each teacher had 52 scores 
corresponding to the 13 competency areas and four types of learning opportunities. We report the 
results of these scores in chapter 3, figure 3.1. 


2. Second, we computed an average frequency of preparation experiences for each of the 13 
competency areas. These scores represent the average frequency of preparation obtained about a 
competency area across the five instructional strategies and four types of learning opportunities. 
In figure E.1, this average is shown as 2.6, which is the average of the 20 ratings in the grid. 
Each teacher had 13 scores corresponding to the 13 competency areas. Because these scores 
represent the average frequency of preparation experiences obtained about a competency area 
regardless of the type of learning opportunity, teachers with the same or similar scores for a 
competency area could have obtained these preparation experiences through different 
frequencies of Coursework, Observation, Practice, and Feedback. We report these scores as the 
sample averages for each competency in chapter 3, figure 3.2. 


3. Third, we computed an average frequency of preparation experiences for each of the four types 
of learning opportunities. These scores represent the average frequency of preparation 
experiences obtained through a learning opportunity across all 65 instructional strategies (five for 
each of 13 competency areas). For example, figure E.1 shows an average frequency of 
preparation experiences obtained through Coursework of 3.6 for the competency area. For each 
teacher, we averaged her/his average frequency score through a learning opportunity across all 
13 competency areas. This resulted in four scores per teacher—one each in Coursework, 
Observation, Practice, and Feedback. Because these scores represent the average frequency of 
preparation experiences obtained through a type of learning opportunity regardless of the 
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competency area, teachers with the same or similar scores for a learning opportunity could have 
obtained these preparation experiences through different frequencies of the 13 competency areas. 
The average frequencies for the types of learning opportunities do not, in and of themselves, 
provide any information about how the frequency of any one of the types of learning 
opportunities is related to the frequencies of the other types of learning opportunities. We report 
these scores as the sample averages for each type of learning opportunity in chapter 3, figure 3.5. 


Figure E.1: Average Frequencies of Preparation Experiences Obtained About a Competency Area 
Through Four Types of Learning Opportunities 


FREQUENCY OF PREPARATION EXPERIENCES 
(where 1—Rarely/Never and 5—Very Often) 


PREPARATION EXPERIENCES THROUGH 


foto} U3) -=\''fo) >3 Game} -3-1 1 a's Were) | PRACTICE FEEDBACK 


STRATEGY 1 
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STRATEGY 3 


STRATEGY 4 
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Table E.1: Measures of Teacher Preparation Experiences Derived From Exploratory Factor 
Analyses 


Ratings of Preparation Experiences With Two Broad Categories of Preparation Experiences 


Calculation Method: 4 ratings (one for each type of learning opportunity) for each of the 2 broad categories 
Creating a Productive Learning Environment—Coursework 
Creating a Productive Learning Environment—Observation 
Creating a Productive Learning Environment—Practice 
Creating a Productive Learning Environment—Feedback 
Promoting Analytic Thinking Skills—Coursework 
Promoting Analytic Thinking Skills—Observation 
Promoting Analytic Thinking Skills—Practice 
Promoting Analytic Thinking Skills—Feedback 
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Appendix F. Teachers’ Assessment of the Usefulness of Preparation 


Experiences 


The teacher survey asked teachers to rate how useful their preparation experiences with each of five 
instructional strategies for each of the 13 competency areas (65 strategies total) had been now that they 
were in the classroom‘ (table F.1). We conducted analyses of teachers’ responses to the usefulness of 
their preparation experiences to provide context to the analyses presented in chapter 3 that examine the 
frequency and variation in those experiences. 


Table F.1: Average Measures of Usefulness 


Ratings of Usefulness of Preparation Experiences With 13 Competency Areas 
(Combined Across Types of Learning Opportunities) 


Calculation Method: Average of 5 usefulness ratings for each competency area: 1 rating for each of 5 instructional 
strategies representing that competency area 


Usefulness of preparation experiences received about Facilitating Extended Classroom Discussions 
Usefulness of preparation experiences received about Maintaining A Positive Classroom Climate 
Usefulness of preparation experiences received about Productive Use Of Time In The Classroom 
Usefulness of preparation experiences received about Demonstrating Sensitivity To Student Needs 
Usefulness of preparation experiences received about Conveying The Importance Of Learning 


Usefulness of preparation experiences received about Managing Student Behavior To Maximize Learning Time 
Usefulness of preparation experiences received about Building Comprehension Of Academic Concepts 
Usefulness of preparation experiences received about Providing Feedback That Helps Student Learning 
Usefulness of preparation experiences received about Building Students’ Higher-Order Thinking Skills 
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Usefulness of preparation experiences received about Designing And Using Assessments Of Student Learning 
Usefulness of preparation experiences received about Effective English Language Arts Instruction 


Bl 
me 


Usefulness of preparation experiences received about Effective Mathematics Instruction 
Usefulness of preparation experiences received about Effective Instruction For English Learners 


an 
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Teachers ranked the usefulness of each instructional strategy on a five-point scale, where 1 = “Have not 
used,” 2 = “Not useful,” 3 = Somewhat useful,” 4 = “Useful,” and 5 = “Very useful” (see figure 2.7 in 
chapter 2 for an example set of survey items, for the competency Maintaining a Positive Classroom 
Climate). The average rating of usefulness for a competency area is the average of the usefulness ratings 
across the five instructional strategies for that competency area. 


Teachers reported that their preparation experiences were at least somewhat 
useful, on average. 


Average usefulness ratings reported by teachers across the 13 competency areas were greater than 3 on 
the five-point scale (see figure F.1). Maintaining a Positive Classroom Climate had the highest average 
rating (4.2) and Effective Instruction for English Learners had the lowest average rating (3.1). 


4 The survey asked teachers to rate the usefulness of each instructional strategy within a competency area without 


considering the type of learning opportunity. 
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A majority of teachers rated their preparation experiences with all but one 
competency area as more than somewhat useful. The exception was Effective 
Instruction for English Learners. 


Between 88 and 67 percent of teachers—yielded by summing the two rightmost sections of each bar in 
figure F.1—reported a rating greater than 3 on the five-point scale. Only 47 percent of teachers rated the 
competency area Effective Instruction for English Learners as more than somewhat useful. In fact, 28 
percent of teachers rated this competency area as not used or not useful. 


Figure F.1: Teacher-Reported Average Usefulness Ratings of Preparation Experiences by Competency Area 


Maintaining a Positive Classroom Climate 31 Sample Avg = 4.2 
Managing Student Behavior to Maximize Learning Time Sample Avg = 4.0 
Productive Use of Classroom Time Sample Avg = 4.0 
Conveying the Importance of Learning Sample Avg = 3.9 
Demonstrating Sensitivity to Student Needs Sample Avg = 3.9 
Designing and Using Assessments of Student Learning Sample Avg = 3.8 
Effective English Language Arts Instruction 


Building Students’ Higher-Order Thinking Skills 


Sample Avg = 3.8 
Sample Avg = 3.8 


Competency Area 


Facilitating Extended Classroom Discussions Sample Avg = 3.8 


Providing Feedback that Helps Student Learning Sample Avg = 3.7 


Effective Mathematics Instruction Sample Avg = 3.6 


Building Comprehension of Academic Concepts Sample Avg = 3.6 


Effective Instruction for English Learners Sample Avg = 3.1 


Percentage of Teachers 


m1 (Not Used) < Avg $ 2 (Not Useful) ™2 (Not Useful) < Avg $ 3 (Somewhat Useful) 3 (Somewhat Useful) < Avg < 4 (Useful) = 4 (Useful) < Avg < 5 (Very Useful) 


NOTE: Teachers rated the usefulness of each of a competency area’s five instructional strategies across all four types of learning opportunities. Responses ranged from 1 to 5, where 
1 = “Have not used,” 2 = “Not useful,” 3 = “Somewhat useful,” 4 = “Useful,” and 5 = “Very useful.” The average usefulness rating for each competency area is the average of the 
usefulness ratings across the five instructional strategies for that competency area. This figure shows the proportion of teachers in four groups based on their average usefulness 
rating for each competency area. Sample size varied between 3,254 and 3,287 due to nonresponse. 

FIGURE READS: The average usefulness of preparation experiences with Maintaining a Positive Classroom Climate was greater than or equal to 1 and less than or equal to 2 on the 
five-point scale for 2 percent of teachers. 

SOURCE: Study’s teacher survey data, 2015. 


Teachers’ usefulness ratings for each of the 13 competency areas were strongly 
related to their ratings of the frequency of those preparation experiences. 


The correlations of the frequency of preparation experiences with each of the 13 competency areas ranged 
from .78 to .85, with the lowest correlation for Managing Student Behavior to Maximize Student 
Learning and the highest for Effective Instruction for English Learners (table F.2). 


Table F.2: Correlations Between Teacher-Reported Usefulness Ratings and Ratings of the 
Frequency of Their Preparation Experiences With the 13 Competency Areas (in Order of 
Increasing Correlation) 


Correlation with Teachers’ 
Usefulness Ratings 

Competency Area 

Managing Student Behavior to Maximize Learning Time 18" 
Maintaining a Positive Classroom Climate 19" 
Productive Use of Classroom Time 19" 
Conveying the Importance of Learning 81" 
Demonstrating Sensitivity to Student Needs 82" 
Designing and Using Assessments of Student Learning 82" 
Effective English Language Arts Instruction 84" 
Building Students’ Higher-Order Thinking Skills 83" 
Facilitating Extended Classroom Discussions 82" 
Providing Feedback That Helps Student Learning 84" 
Effective Mathematics Instruction 85" 
Building Comprehension of Academic Concepts 83" 
Effective Instruction for English Learners 85" 

** oO < 01. 


NOTE: Competency area ratings are an average of teachers’ rating across all four types of learning opportunities (Coursework, 
Observation, Practice, and Feedback). 

TABLE READS: The correlation between the average usefulness rating and the average frequency rating of preparation 
experiences with Maintaining a Positive Classroom Environment was .79. 

SOURCE: Study’s teacher survey data, 2015. 


Teachers tended to rate usefulness higher for competency areas for which they reported more preparation 
experiences (figure F.2). This could indicate that the practices that preparation programs focus on are 
subsequently more useful to teachers in the classroom than are less-emphasized practices. However, it 
also could reflect a reporting bias whereby teachers tended to use the same rating for frequency and 
usefulness when considering their preparation experiences with a competency area. 


We did not see any evidence that usefulness ratings varied among teachers with similar ratings of 
frequency of preparation experiences. That is, teachers who reported receiving more frequent preparation 
experiences for a competency area also consistently rated those preparation experiences as more useful. In 
other words, there was little variation in the usefulness ratings to suggest that some teachers were getting 
more frequent preparation experiences but the experiences were less highly useful or less high-quality and 
ultimately might not be as likely to be positively related to teaching effectiveness. 
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Figure F.2: Means and 95 Percent Confidence Intervals for Teacher-Reported Usefulness Ratings 
and Ratings of the Frequency of Their Preparation Experiences With the 13 Competency Areas 
(in Order of Increasing Usefulness) 
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Building Students' Higher-Order Thinking Skills 


es , F Usefulness 
Facilitating Extended Classroom Discussions Experiences 


Providing Feedback that Helps Student Learning Usefulness 


Effective Mathematics Instruction Sorell 
Building Comprehension of Academic Concepts senha 
Usefulness 


Effective Instruction for English Learners Experiences 


2.5 3 3.5 4 45 


Usefulness Rating/Frequency of Preparation Experiences 


NOTE: Teachers’ responses to the survey questions about usefulness of preparation experiences ranged from 1 to 5, where 1 = 
“Have not used” and 5 = “Very useful.” Teachers’ responses to the survey questions about frequency of preparation experiences 
ranged from 1 to 5, where 1 = “Rarely/Never” and 5 = “Very often.” The 13 competency areas are listed in order of highest to lowest 
average score. Sample size varied between 3,249 and 3,289 due to nonresponse. 

FIGURE READS: The average rating of usefulness of preparation experiences with Maintaining a Positive Classroom Climate was 
4.2. The average rating of the frequency of preparation experiences with Maintaining a Positive Classroom Climate was 4.0. 
SOURCE: Study’s teacher survey data, 2015. 
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Appendix G. Factor Analysis Methodology 


The methodology for creating the two broad categories of preparation experiences was a two-stage 
exploratory factor analysis. In the first stage, we examined the factor structure among the 65 individual 
survey items that comprised the 13 competency areas separately within each of the four types of learning 
opportunities. The first-stage exploratory factor analysis reduced the set of 65 individual strategies to a set 
of 11 competency areas.” We created factor scores for the 11 competency areas using SAS Proc Factor 
(version 9.4) after applying an oblique (promax) rotation. This process creates factor scores as weighted 
means of the individual strategy items, where weights are proportional to the factor loadings. We 
calculated Cronbach’s alpha to measure the internal reliability of the strategies that made up each 
competency area.° The alpha coefficients ranged from 0.87 to 0.97 across the 44 competency area 
measures (11 competency area measures within each of the four types of learning opportunities). 


The second-stage analysis reduced the set of 11 competency areas within each type of learning 
opportunity to two broad categories (see table G.1). Of the 11 competency areas, 8 loaded heavily on one 
or the other of the two broad categories for all four types of learning opportunities (Coursework, 
Observation, Practice, Feedback). We defined “heavily” as when a competency area had factor loadings 
that were larger than 0.40 and were more than 1.5 times as large as their loadings for the other broad 
category. The three remaining competency areas loaded heavily on one of the two broad categories for 
only one or two of the types of learning opportunities (and did not load heavily on the other category for 
any of the types of learning opportunities). Specifically, Demonstrating Sensitivity to Student Needs 
loaded heavily on creating a productive learning environment only for Coursework; Facilitating 
Extended Classroom Discourse loaded heavily on promoting analytic thinking skills only for Observation 
and Practice; and Designing and Using Assessments of Student Learning loaded heavily on promoting 
analytic thinking skills only for Coursework and Practice (see table G.2). 


We created factor scores for the two broad categories within each type of learning opportunity as 
described for the first-stage analysis. For each of the broad categories, we calculated a factor score by 
weighting the means of the 11 competency areas proportional to their factor loadings. For ease of 
interpretation, we re-scaled the factor scores of the two broad categories to make them roughly equivalent 
to the scale of the original strategy items that ranged from 1 = “Rarely/Never” to 5 = “Very often.” Re- 
scaling had no effect on the inferences from the relational analyses relating preparation experiences to 
teaching effectiveness, but re-scaling facilitated interpretation of the regression coefficients from those 
relational models. Cronbach’s alpha ranged from 0.96 to 0.98 across the eight broad category measures 
(two measures within each of four types of learning opportunities). 


Decisions regarding the number of factors to retain were based on results from the revised version of Velicer’s 
MAP test described in O’ Connor (2000). 


Each strategy loaded most highly onto a single competency area. We define the set of strategies that make up a 
competency area as the set that has their largest factor loading on that competency area. 
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Table G.1: Loading of 11 Competency Areas on Two Broad Categories 


Two Broad Categories From Second-Stage Factor Analysis 


Creating a Productive 
Learning Environment 


Promoting Analytic 
Thinking Skills 


Of the 11 competency areas from the 
first-stage factor analysis, 8 loaded 
heavily on only one of the two broad 
categories across all four types of 
learning opportunities 


1) Maintaining a Positive Classroom 
Climate 


2) Managing Student Behavior to 
Maximize Learning Time 


3) Productive Use of Classroom Time 


4) Conveying the Importance of 
Learning 


5) Building Comprehension of Academic 
Concepts / Building Students’ Higher- 
Order Thinking Skills / Providing 
Feedback That Helps Student 
Learning 

6) Effective ELA Instruction 

7) Effective Math Instruction 

8) Effective Instruction for English 
Learners 


Of the 11 competency areas from the 
first-stage factor analysis, 3 loaded 
heavily on the two broad categories 
across the four types of learning 
opportunities 


9) Demonstrating Sensitivity to 
Student Needs (Coursework only) 


10) Facilitating Extended Classroom 
Discussions (Observation, Practice) 

11) Designing and Using Assessments of 
Student Learning (Coursework, 
Practice) 
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Table G.2: Items and Loading Used to Create Two Broad Categories Within Each Type of Learning Opportunity 


Coursework Observation Practice Feedback 
Creating a Promoting Creating a Promoting Creating a Promoting Creating a Promoting 
Productive Analytic Productive Analytic Productive Analytic Productive Analytic 
Learning Thinking Learning Thinking Learning Thinking Learning Thinking 

Competency Area Environment Skills Environment Skills Environment Skills Environment Skills 
Maintaining a Positive 7 a a " 
Claseroora Climate 0.64 0.10 0.69 0.07 0.69 0.06 0.63 0.13 
Managing Student 
Behavior to Maximize 0.55* 0.16 0.66* 0.08 0.68* 0.08 0.58* 0.17 
Learning Time 
ee 0.73* 0.03 0.71* 0.05 0.61* 0.10 0.60* ~0.01 
Classroom Time 
Seng Me mpanente 0.58* 0.15 0.53* 0.22 0.55* 0.21 0.52* 0.23 
of Learning 
Demonstrating Sensitivity rs Z . é 
to Student Needs 0.51 0.15 0.27 0.33 0.27 0.32 0.38 0.29 
EAbinaung Eeenuee 0.31 0.35* 0.27 0.41* 0.17 0.47* 0.35* 0.34 
Classroom Discussions 
Designing and Using 
Assessments of Student 0.20 0.49* 0.35 0.35* 0.34* 0.34 0.26 0.41* 
Learning 
Building Students’ Higher- 
Order Thinking Skills / 
Providing Feedback That 0.25 0.62* 0.24 0.65* 0.21 0.67* 0.18 0.66* 
Helps Student Learning / 
Building Comprehension 
of Academic Concepts 
Effective ELA Instruction 0.10 0.64* 0.24 0.52* 0.22 0.54* 0.17 0.57* 
Effective Math Instruction 0.09 0.54* 0.20 0.49* 0.21 0.46* 0.14 0.53* 
Eee ve IistiGeuon ior 0.04 0.50* 0.06 0.61* 0.02 0.59* 0.01 0.58* 
English Learners 


NOTE: Loading “heavily” is defined as when a competency area had factor loadings that were larger than 0.40 and were more than 1.5 times as large as their loadings for the other 


broad category. 


Key 


Competency loads “heavily” on creating a productive learning environment 


Competency loads “heavily” on promoting analytic thinking skills 
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Appendix H. Assessing Measurement Error in Teacher Survey 


Responses 


In this section, we present more detailed results of analyses that examined variation in preparation 
experiences associated with teachers’ route to certification, which we conducted to help assess the 
measurement error in the survey. We expected that teachers certified through traditional routes would 
report more preparation experiences than would teachers certified through alternative routes, since 
teachers certified through traditional routes generally spend more time in their preparation programs and 
complete their programs prior to becoming teachers of record. If teachers’ reported experiences are 
related to providers, it supports the assumption that the observed variation in teachers’ survey is 
systematic and not solely the result of measurement error. Variation that arises from measurement 
error would not be correlated with a teacher’s provider. The analyses showed that teachers prepared 
through traditional routes to certification reported more preparation experiences with 12 of the 13 
competency areas than did teachers certified through alternative routes to certification, which contributed 
to our confidence that the variation in teacher responses was systematic. Our approach to and results of 
the analysis are described below. 


We compared the preparation experiences of teachers in traditional routes to certification, alternative 
routes to certification excluding Teach For America (TFA), and TFA.’ We separated TFA from other 
alternative routes to certification because it has been the most studied alternative route and has been 
shown to be associated with higher student achievement, specifically in math (Clark et al. 2013, 2015; 
Decker, Mayer, and Glazerman 2004).° 


Teachers prepared through traditional routes to certification reported more frequent preparation 
experiences with 12 of the 13 competency areas than did teachers certified through alternative routes to 
certification. For example, as shown in figure H.1, teachers certified through traditional routes reported an 
average rating for Maintaining a Positive Classroom Climate of 4.1, compared to a rating of 3.5 and 3.4 
for those certified through non-TFA and TFA alternative routes, respectively.’ The frequency of 
preparation experiences of the two groups of teachers trained through alternative routes to certification 
were not significantly different from each other for 7 of the 13 competency areas. The difference between 


7 The study sample included 2,578 teachers from traditional routes to certification, 628 teachers from non-TFA 
alternative routes to certification, and 85 teachers from TFA. Information on route to certification was not 
available for three teachers. We tested the statistical significance of the differences in the experiences measures 
across the three groups, as well as the differences between any two of the groups, using two-sided tests that 
accounted for the clustering of teachers within sites. See table H.1 for the p-values from these tests. 


The differences in the effectiveness of non-TFA teachers and their TFA colleagues could reflect differences in 
their preparation as well as differences in individual characteristics, which could be the result of the recruitment 
process used by TFA. 


9 For 12 of the 13 competency areas, the differences between the preparation experiences of teachers from 
traditional routes to certification and teachers from non-TFA or TFA alternative routes to certification were 
statistically significant at the p < .05 level (all values p < .01). The exception was the difference between 
teachers from traditional routes to certification and TFA teachers for the competency area Managing Student 
Behavior (p = .06). 
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the two groups was the largest for Effective Instruction for English Learners (average 2.6 for non-TFA 
and 2.1 for TFA alternative certification teachers). '° 


10 This difference was statistically significant (p < .01). Differences between the two groups were also statistically 


significant for Building Comprehension of Academic Concepts (p < .01), Effective Mathematics Instruction 
(p = .04), Providing Feedback That Helps Student Learning (p = 0.03), Facilitating Extended Classroom 
Discussions (p = .01), and Demonstrating Sensitivity to Student Needs (p = .01). 


Figure H.1: Teacher-Reported Frequency of Preparation Experiences With Competency Areas by Certification Route 
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Instruction 


Classroom Climate 
Learning 


Maintaining a Positive 
Learners 


Student Needs 
Designing and Using 
Assessments of Student 

Effective Mathematics 


Managing Student Behavior to 
Maximize Learning Time 
Productive Use of Classroom 
Conveying the Importance of 
Learning 
Demonstrating Sensitivity to 
Effective English Language Arts 
Instruction 
Building Students’ Higher-Order 
Thinking Skills 
Facilitating Extended Classroom 
Discussions 
Providing Feedback that Helps 
Student Learning 
Building Comprehension of 
Academic Concepts 
Effective Instruction for English 


Competency Area 


* indicates that the difference in the preparation experiences of teachers from traditional routes to certification and teachers from TFA was statistically significant at the p < .05 level. 

t indicates that the difference in the preparation experiences of teachers from traditional routes to certification and teachers from non-TFA alternative routes to certification was 
statistically significant at the p < .05 level. 

* indicates that the difference in the preparation experiences of teachers from non-TFA alternative routes to certification and TFA teachers was statistically significant at the p < .05 
level. 

NOTE: Teachers’ responses to the survey questions about preparation experiences ranged from 1 to 5, where 1 = “Rarely/Never” and 5 = “Very often.” Traditional refers to teachers 
prepared through traditional routes to certification, Non-TFA Alternative Route refers to teachers prepared through all alternative routes to certification except Teach For America, and 
TFA refers to teachers prepared through Teach For America. Sample size for teachers prepared through traditional routes varied from 2,545 to 2,575 due to nonresponse. Sample size 
for teachers prepared through non-TFA alternative routes to certification varied from 618 to 627. Sample size for TFA varied from 82 to 85. 

FIGURE READS: For Maintaining a Positive Classroom Climate, the sample average of preparation experiences was 4.1 for teachers prepared through traditional routes to 
certification, 3.5 for teachers prepared through non-TFA alternative routes, and 3.4 for TFA teachers. The difference between teachers prepared through traditional routes and 
teachers from either of the two alternative routes was statistically significant at the p < .05 level. 

SOURCE: Study’s teacher survey data, 2015. 
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Teachers prepared through traditional routes to certification reported significantly more frequent 
preparation experiences through each of the four types of learning opportunities than those prepared 
through alternative routes to certification. For teachers prepared through traditional routes to certification, 
averages on the four types of learning opportunities ranged from 3.4 to 3.8, whereas for teachers from 
non-TFA and TFA alternative routes to certification, averages ranged from 2.9 to 3.6. 


Comparing teachers prepared through non-TFA and TFA alternative routes to certification, teachers from 
TFA reported significantly fewer experiences obtained through Coursework and Observation than did 
teachers from non-TFA alternative routes to certification. Teachers from TFA and teachers from non-TFA 
routes to certification reported frequencies of experiences obtained through Practice and Feedback that 
did not differ significantly (figure H.2). For detailed results of these analyses, see table H.1. 


Figure H.2: Teacher-Reported Frequency of Preparation Experiences Through Types of Learning 
Opportunities by Certification Route 
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* indicates that the difference in the preparation experiences of teachers from traditional routes to certification and teachers from 
TFA was statistically significant at the p < .05 level. 

t indicates that the difference in the preparation experiences of teachers from traditional routes to certification and teachers from 
non-TFA alternative routes to certification was statistically significant at the p < .05 level. 

* indicates that the difference in the preparation experiences of teachers from non-TFA alternative routes to certification and TFA 
teachers was statistically significant at the p < .05 level. 

NOTE: Teachers’ responses to the survey questions about preparation experiences ranged from 1 to 5, where 1 = “Rarely/Never” 
and 5 = “Very often.” Traditional refers to teachers prepared through traditional routes to certification, Non-TFA Alternative Route 
refers to teachers prepared through all alternative routes to certification except Teach For America, and TFA refers to teachers 
prepared through Teach For America. Sample size for traditional routes to certification varied from 2,573 to 2,574. Sample size for 
non-TFA alternative routes to certification varied from 625 to 626. Sample size for TFA was 85. 

FIGURE READS: For coursework, the sample average of preparation experiences was 3.8 for teachers from traditional routes to 
certification, 3.6 for teachers from non-TFA alternative routes to certification, and 3.4 for TFA teachers. The difference between 
teachers prepared through traditional routes and teachers from either of the two other routes was statistically significant at the 
p<.05 level. 

SOURCE: Study’s teacher survey data, 2015. 
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Table H.1: Descriptive Statistics of Preparation Experiences With Competency Area and Type of Learning Opportunity by Certification 
Route 


Competency Area 

Melitainindia Ses 4,07 0.74 3.54 0.98 3.44 0.79 | <.001 | <.001 053 < 001 
Classroom Climate 

Managing) SueienE Benavier i 3.87 0.84 3.51 1.01 3.66 os9 | <0 | <.001 101 < 001 
Maximize Learning Time 

Stee Hee 0} Classtepin 3.84 0.82 3.50 1.01 3.52 075 | <.001 | <.001 863 < 001 
Carne yiigy Ne Mponarinert 3.79 0.86 3.39 1,02 3.48 0.93 | <.001 | <.001 113 < 001 
Learning 

Demonsttanng Sensiiviy 13 3.76 0.89 3,30 1.05 2.96 092 | <.001 | <o01 | <2 < 001 
Student Needs 

Designing and Using 

Assessments of Student 3.69 0.91 3.17 1.06 3.12 089 | <.001 | <.001 438 <.001 
Learning 

Elfecive Gnulishy Language Pils 3.67 0.94 3.08 111 3.07 099 | <.001 | <.001 833 < 001 
Instruction 

Building Students’ Higher-Order 3.64 0.90 3.15 1.02 3.25 092 | <.001 | <.001 071 < 001 
Thinking Skills 

Faclitaing Exended 3,62 0.86 3,22 0.97 3.10 os1 | <0 | <.001 013 < 001 
Classroom Discussions 

Piowidingi-Cedbaek iia Fle|ps 3.54 0.91 3.10 1.03 2.90 os7 | <0 | <.001 | <.001 < 001 
Student Learning 

eee Mancmale 3.52 1.09 2.91 1.20 2.62 119 | <0. | <.001 | <.001 < 001 
Instruction 


peemreniioal ot 3.47 0.91 3.01 1.02 2.84 087 | <.001 | <.001 002 < 001 
aed Insutuction-ior English 2.89 1.23 2.61 1.17 2.13 107 | <.001 | <.0o1 | <.001 <.001 
Type of Learning Opportunity 

Coursework 3.82 0.77 3.56 0.85 3.40 072 | <.001 | <.001 | <.001 < 001 
Observation 3.67 0.77 3.20 1.00 2.94 07s | <oo | <oo | <.001 < 001 
Practice 3.64 0.78 2.94 1.16 2.97 081 | <.001 | <.001 737 < 001 
Feedback 3.44 0.91 3.05 1.07 3.03 081 | <.001 | <.001 609 < 001 


NOTE: Teachers’ responses to the survey questions about preparation experiences ranged from 1 to 5, where 1 = “Rarely/Never” and 5 = “Very often.” Sample sizes for teachers from 
traditional routes to certification varied from 2,545 to 2,574. Sample sizes for teachers from alternative routes to certification (excluding Teach For America) varied from 615 to 627. 
Sample sizes for Teach For America varied from 82 to 85. Tests for statistical significance are clustered at the district or state level. 

TABLE READS: The sample mean of Maintaining a Positive Classroom Climate was 4.07 with a standard deviation of 0.74 for teachers from traditional routes to certification. For 
alternative routes to certification (excluding Teach For America), the sample mean was 3.54 and the standard deviation was 0.98. For Teach For America, the sample mean was 3.44 
and the standard deviation was 0.79. The test that the mean of Maintaining a Positive Classroom Climate for teachers from traditional routes to certification was equal to teachers from 
alternative routes to certification (excluding Teach For America) had p < .001. The test that the mean of Maintaining a Positive Classroom Climate for teachers from traditional routes to 
certification was equal to teachers from Teach For America had p < .001. The test that the mean of Maintaining a Positive Classroom Climate for teachers from alternative routes to 
certification (excluding Teach For America) was equal to teachers from Teach For America had p = 0.053. The test that all three means were equal had p < .001. 

SOURCE: Study’s teacher survey data, 2015. 
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In this appendix, we first describe how the measures of effectiveness were created, and then describe the 
regression modeling approach to estimating the relationships of preparation experiences to effectiveness. 


Section I.1: Measures of Effectiveness 


In order to estimate teaching effectiveness, student achievement outcomes were obtained from 
administrative data linking student-level ELA and math state achievement test results to the teacher or 
teachers who were responsible for students’ grade 4, 5, or 6 ELA or math instruction in the 2012-13, 
2013-14, or 2014-15 school year.'! Student records were linked over time and went back to the prior year 
(2011-12) such that baseline (pre-test) measures of ELA and math achievement and baseline student 
demographic measures were available for each student. The student-teacher linked data were obtained 
from 19 separate “sites” in 14 states. Each “site” was a state or district educational agency that was able to 
provide the study with student-teacher linked data for all students and teachers in the relevant grades and 
years.'* We use the term “site” going forward to reference the state or district in which teachers were 
clustered for analysis. 


Teaching effectiveness was estimated separately for (1) effectiveness of ELA instruction for all of a 
teacher’s students; (2) effectiveness of ELA instruction for a teacher’s EL students (if she/he had at least 
one EL student); (3) effectiveness of math instruction for all of a teacher’s students; and (4) effectiveness 
of math instruction for the teacher’s EL students (if she/he had at least one EL student). Each of these four 
types of teaching effectiveness was estimated separately within sites, grades, and school years. When a 
teacher taught students in multiple grades in a single school year, his/her multiple effectiveness measures 
across grades were combined as a weighted mean into a single measure for that school year. When a 
teacher had effectiveness measures from multiple years—for example, from his/her first, second, and/or 
third year of teaching—those were retained as separate measures such that relationships of preparation 
experiences to teaching effectiveness could be examined for first-, second-, and third-year teachers. 


Effectiveness measures were calculated for all teachers within each site who taught the relevant grades in 
the relevant years. Those measures were standardized such that, within each site, the distribution of 
effectiveness scores had a mean of 0 and a standard deviation of 1. This means that a teacher who was 
equally effective as the average teacher within his/her site would have an effectiveness score of 0. A 
teacher with an effectiveness score of, say, 0.50 was more effective than the average teacher within the 
site by 0.50 standard deviation units. A score of, say, -0.50 means that the teacher was less effective than 
the average teacher in that site by 0.50 standard deviation units. A negative score indicates that a teacher’s 


11 When two (or more) teachers were responsible for a student’s instruction in one subject (either ELA or math), 


the student’s achievement contributed to the effectiveness measures of both (or all) teachers who were 
responsible for instruction. For example, if two teachers were responsible for the student’s ELA instruction, 
then each of the two teachers received half of the credit for that student’s achievement growth (or more 
accurately, for the difference between the student’s observed and predicted math achievement score). 


12 Surveys regarding preparation experiences were administered to teachers from 30 sites, but due to 


considerations of cost and expected numbers of surveyed teachers in each site, we limited the collection of 
student-teacher linked achievement data to the 19 largest sites. 
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students performed worse on their end-of-year assessments than did similar students taught by different 
teachers in the district or state. 


The effectiveness scores of the surveyed teachers from each site were extracted from the larger sets of 
effectiveness scores of all teachers within the sites, and subsequently were merged to the survey data on 
preparation experiences. The merged dataset was used to estimate relationships between preparation 
experiences and teaching effectiveness (appendix I.2). The measures of teaching effectiveness are 
estimates that were obtained from fitting statistical models to student-teacher linked data. We will refer to 
the statistical models that produced the estimates of effectiveness as “level-1 models.” A second set of 
statistical models, which we refer to as “level-2 models” was used to estimate the relationships between 
preparation program experiences and teaching effectiveness. 


The remainder of this section provides additional details on the estimation of teaching effectiveness 
scores (the level-1 models). Appendix section I.2 provides details on the level-2 models. 


Students Who Link to More Than One Teacher 


A fundamental challenge in estimating the effect that a teacher has on the achievement of his/her students 
is that, in some cases, more than one teacher is responsible for the math or reading instruction of a 
particular student. To account for shared teaching responsibility, analyses were conducted in several 
steps. First, a cell identifier (“cell-ID”) was created for each unique combination of links of teachers to 
students. In the first sub-step of the “level-1 model,” student achievement was modeled as the dependent 
variable, with pre-tests and other covariates on the right-hand side of the regression model, and dummy 
variables for each of the cell-IDs. The model produces an estimate (and standard error) of the level-1 cell 
effect for each cell-ID. In a subsequent sub-step, teacher level-1 scores were calculated as weighted 
means of the cell effect scores to which the particular teacher contributed. 


The table below illustrates example cells for grade 5 math students who were assessed in the spring of 
2014. The “cell” in row 1 represents 18 students who were taught math by a single teacher (Teacher ID 
=T1). The 27 students in row 2 were taught by two teachers (Teacher IDs T2 and T3). In the model 
shown in equation 1, dummy variables for cell IDs are included in the model. 


Table 1.1: Example Cells for Grade 5 Math Students 


Row Teacher ID Cell-ID COUNT 
1 T1 Cell 1 18 
2 T2, T3 Cell 4 27 


Model Used to Estimate Level-1 Cell Effects 


The first step of the level-1 analysis will produce a cell value added estimate (and standard error) for each 
cell using the following prototypical model specification: 


(Yic) = a wp, Cellic a0 rneien i) + Eic (Eqn. 1) 


where: Y;, is the spring reading/ELA or math achievement test score from i” student (i in 1,2,...,1) of the 
c™ cell (c in 1,2,...,C cells); Cell;, is the indicator variable for the c" cell (i.e., the fixed effect for cells); 
X?t is the n™ student-level covariate, which included prior-year test scores in both math and reading/ELA, 
gender, race/ethnicity, English learner status, age, FRPL status, and other student characteristics available 
in the administrative database of a given site. 
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The outputs equation 1 model that were used in subsequent steps were the level-1 cell effectiveness 
estimates w, ... Yc; and the standard error estimates of the level-1 cell estimates SE(;) ... SE(q,). 


Calculating Level-1 Teacher Estimates From Level-1 Cell Estimates 


The level-1 teacher scores were calculated as weighted means of all level-1 cell scores to which a teacher 

contributed. 

B, = Zeaabewe (Eqn. 2) 
J yee We : 

where the weights, w,, are instructional responsibility weights and are equal to the number of students in 

the cell divided by the number of teachers in the cell. The outputs from this step were the level-1 teaching 

effectiveness scores 6, se Bi: and their standard errors SE(6, yi. SE(f;). 


Standardizing the Level-1 Teacher Estimates 


The “raw” level-1 teacher estimates yielded by the steps above (B, oe By) were subject to an adjustment 
that was motivated by the observation that teaching effectiveness estimates tend to be more variable in 
elementary grades than in middle school grades (Koedel, Mihaly, and Rockoff 2015). At face value, this 
suggests that differences in true teaching effectiveness are larger at earlier, rather than later, grades. 
However, it is plausible that differences in the variances of teacher effects are an artifact of the procedures 
used to convert raw test scores into scale scores (Morgan et al. 2004). To address this issue, level-1 
teacher scores were standardized for each subject (math or reading/ELA), administrative data source 
(state or district), grade level (4, 5, 6), and school year (2012-13, 2013-14, 2014-15) such that the 
variance and thus the range of level-1 teacher scores are the same for all grade levels in all districts in all 
years. That standardization and calculation of the standard errors of the standardized level-1 scores are 
described below. 


This standardization step was done within administrative data source (district or state), subject (math or 
reading/ELA) and grade level (4, 5, 6), and school year (2012-13, 2013-14, 2014-15). The standardized 
scores were calculated as 


5 _ Bi-Bw 


44 ~ watsD(Bp (Eqn. 3) 


where B; = the level-1 estimate for teacher j; Bu = the mean of the level-1 estimates for teachers weighted 


by the number of students taught by teacher j; and wgtSD(f) is the standard deviation of the level-1 
teachers scores, weighted by the number of students taught by teacher j. 


The outputs from this step that were used in the level-2 model (described in appendix section I.2) were: 
Zjagy — the standardized measure of teaching effectiveness for the j" teacher in the d" administrative data 
source in the g" grade in the y" year; and SE(Zjagy) = the standard error of the standardized measure of 
teaching effectiveness. 
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Section I.2: Estimating the Relationships of Preparation Experiences to 
Effectiveness 


Relational analyses were conducted to address the following questions concerning the relationships of 
preparation experiences to math or ELA teaching effectiveness of grade 4—6 teachers who were in their 
first, second, or third year of teaching: 


e What, if any, preparation experiences are related to teaching effectiveness for all students? 


e Do relationships of experiences and effectiveness diminish (or increase, or stay the same) as teachers 
gain more experience in the classroom? 


The dependent variables in the regression models were estimates of teaching effectiveness, and the 
estimates for teachers with greater numbers of students are more precise than estimates for teachers with 
fewer students. Thus, the regression models used to estimate the relationships between experiences and 
effectiveness used regression weights such that teachers with more-precise estimates of effectiveness 
were given greater weight than those with less-precise estimates. ® 


The models include indicators for the sites (state or school districts) from which the measures of 
effectiveness were estimated, and controls for whether teachers were in their first, second, or third year of 
teaching at the time their effectiveness was measured. The site indicators were a crucial part of the models 
because the measures of effectiveness are relative to other teachers within the site. This means that an 
average teacher in Site A is more effective than a below-average teacher in Site A, and an average teacher 
in Site B is more effective than a below-average teacher in Site B, but what the models do not tell us is 
whether an average teacher at Site A is more, less, or equally as effective as an average teacher in Site B. 


By including indicator variables for sites as covariates in the models, we ensure that the relationships 
between preparation experiences and effectiveness are estimated within sites, where relative effectiveness 
is measured, and then that the within-site relationships are averaged across sites to give an overall, 
average measure of the relationships between preparation experiences and effectiveness. The primary 
models also control for years of teaching experience, because tenure was found to have a strong 
relationship to effectiveness in our data and has been consistently shown to be related to teaching 
effectiveness in other studies. ' 


To account for the multiple observations per teacher that occurred when teachers had effectiveness 
measured at more than one time point (in their first, second, and/or third year of teaching), two-level 
hierarchical linear models were fit to the data with repeated observations (level-1) nested in teachers 
(level-2). The regression models were of the form: 


2ij = Bo + ao; + By PrepExp; + B,Yr1jj + B3Yr2ij + LET m,Site? +, (Eqn. 4) 


where, 


13 The weights were calculated using feasible generalized least squares weights as described in Lewis and Linzer 


(2005) and Hanushek (1974). 


The following studies have found years of experience to be related to effectiveness: Boyd et al. (2009), Harris 
and Sass (2007, 2011), Kini and Podolsky (2016), and Xu, Hannaway, and Taylor (2009). 


14 
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Zij = The i" effectiveness measure for the j teacher. Teachers could have up to three 

effectiveness outcomes measures if their effectiveness was measured in their first, 
second, and third year of teaching. Each model had either ELA or math effectiveness 
as the outcome measure. 


Qo; = Arandom intercept term for teacher j. 


PrepExp; = A preparation experience measure for teacher j. Each model included one of the eight 
following preparation experience measures: 


creating a productive learning environment through Coursework 
creating a productive learning environment through Observation 
creating a productive learning environment through Practice 
creating a productive learning environment through Feedback 
promoting analytic thinking skills through Coursework 
promoting analytic thinking skills through Observation 
promoting analytic thinking skills through Practice 

promoting analytic thinking skills through Feedback 


Yriij = A dummy variable taking the value 1 if teaching effectiveness was measured when 
the teacher was in the first year of teaching, and taking the value 0 when the teacher 
was in the second or third year of teaching when effectiveness was measured. 


Yr2;; = A dummy variable taking the value 1 if teaching effectiveness was measured when 
the teacher was in the second year of teaching, and taking the value 0 when the 
teacher was in the first or third year of teaching when effectiveness was measured. 


Site; = Dummy variables for sites taking the value 1 if teaching effectiveness of the j" 
teacher was measured in site s (s = 1,2...,19) and taking the value 0 otherwise. 


ij = Residual error. 


The parameter estimate f, that is produced by the model describes the relationship between preparation 
experiences and teaching effectiveness. The parameter estimate is interpreted as in the following example. 
An estimate 8, = 0.09 indicates that a one-unit increase in the experiences measure (on a five-point scale 
ranging from 1 = Rarely/Never to 5 = Very often) is associated with an increase in teaching effectiveness 
equal to 0.09 standard deviation units of teaching effectiveness. 


To further facilitate the interpretation of parameter estimates, we calculated the average increase in 
teaching effectiveness between first- and second-year teachers and between second- and third-year 
teachers and compared the magnitudes of the increases in teaching effectiveness associated with 
additional preparation experiences versus the increases in teaching effectiveness associated with gaining 
an additional year of teaching experience. 


Models with additional control covariates were also fit to the data to assess whether the results were 
sensitive to the inclusion/exclusion of additional controls for classroom contextual variables and other 
teacher characteristics (see appendix K for descriptions and results of these robustness analyses). 
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Subgroups 

To investigate the relationships between preparation experiences with creating a productive learning 
environment and with promoting analytic thinking skills to teaching effectiveness for subgroups of 
teachers with differing student populations of at-risk students, we added interaction terms 

(PrepExp; * AtRiskStudents;) to the model shown in equation (1). These terms produced estimates of 
the relationships of experiences to effectiveness for teachers who had higher and lower proportions of 
students who were eligible for free or reduced-price lunch, and for teachers who taught students whose 
average baseline performance was in the bottom half or top half of the performance distribution. (See 
appendix L for details and cut-points for the indicators.) 


We conducted additional analyses to estimate the relationships between preparation experiences with 
creating a productive learning environment and with promoting analytic thinking skills to effectiveness of 
teaching English learners. For these analyses, we calculated the measures of effectiveness of teaching 
English learners using only the student-teacher linked data of students who were English learners. The 
method for estimating teaching effectiveness for these students was exactly the same as described above 
in appendix section I.1. Those measures of teaching effectiveness for English learners were the dependent 
measures in the level-2 models, described above in this section. 


More Classroom Experience 


To investigate whether the relationships between preparation experiences and effectiveness diminished 
(or increased, or remained the same) as teachers gained more experience in the classroom, interaction 
terms (PrepExp; * Yr1;; and PrepExp; * Yr2;;) were added to the model shown in equation (1). These 
terms produced estimates of the relationships of experiences to effectiveness when teachers where in their 
first, second, and third years of teaching, and they produced a test of whether the relationships varied by 
years of teaching experience. For results of these analyses, see appendix M. 
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Appendix J. Teachers’ Preparation Experiences by Preparation 
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In this appendix, we examine teacher survey responses by preparation provider and degree program.'” 
With these analyses, we can see how much of the observed variation in preparation experiences 
documented in chapter 3 is at the individual teacher level, even among teachers who attend the same 
preparation provider or are in the same degree program within a program. 


Variation in Preparation Experiences Associated With Preparation Provider and Degree Program 


Within each of the competency areas and across each of the types of learning opportunities, we examined 
teachers’ preparation experiences by preparation providers and by degree programs within a provider. 
Figure J.1 provides an illustrative example based on one of the competency areas (Maintaining a Positive 
Classroom Climate) showing findings for three preparation providers that offer both a bachelor’s (BA) 
and a master’s (MA) degree program. 


We found a pattern of more variation in preparation experiences within 
providers (and degree programs within a provider) than across providers for 
each of the competency areas, as well as for each of the types of learning 
opportunities. 


As can be seen in the figure, teachers’ individual preparation experiences varied widely, ranging between 
1.5 and 5 for Providers 1 and 3 and between 3 and 5 for Provider 2. In contrast, the average of this 
competency area for each of the providers (and degree programs within providers) was very similar— 
between 3.7 and 4.4, as shown by the large black diamonds in the figure. 


We also conducted variance decomposition analyses using preparation experiences responses from all 159 
preparation providers that had at least five responding teachers each. The analysis used the nested 
structure of the data (teachers nested within degree programs within preparation providers'®) and 
estimated hierarchical linear models to decompose the total variance observed in each of the competency 
areas and in each of the types of learning opportunities average experiences measures to components at 
three levels: 


1. Preparation provider level: Percentage of the total variance that was explained by preparation 
providers. The larger this percentage was, the more varied the preparation experiences were between 
the preparation providers represented in the sample. 


15 To ensure some degree of statistical stability in these analyses, we included only the 2,601 teachers from 159 


preparation providers that had at least five responding teachers each. 


16 The analysis was restricted to preparation providers that were represented by five or more teachers in order to 


increase the reliability of estimated variance components. The nesting structure was as follows: For preparation 
providers that offered only traditional routes to certification, we placed teachers into two groups: BA program 
(which included four-year and five-year degree programs) or MA program. For preparation providers that 
offered only alternative routes, we placed teachers into two groups: MA degree program or no (additional) 
degree program. For preparation providers that offered both traditional and alternative routes, we placed 
teachers into one of four groups: BA degree-traditional certification program, MA degree-traditional 
certification, MA degree-alternative certification, or no degree-alternative certification. 
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Figure J.1: Variation in Preparation Experiences With Maintaining a Positive Classroom Climate 
Within Degree Programs for Three Preparation Providers 
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NOTE: This figure shows the actual average frequency of preparation experiences (y-axis) for the competency area Maintaining a 
Positive Classroom Climate. The graph added jitter on the x-axis to distinguish teachers who had similar/very close average 
preparation experiences. Preparation Provider 1 accounted for 91 teachers (72 in its bachelor’s program and 19 in its master’s 
program). Preparation Provider 2 accounted for 33 teachers (16 in its bachelor’s program and 17 in its master’s program). 
Preparation Provider 3 accounted for 38 teachers (21 in its bachelor’s program and 17 in its master’s program). 

FIGURE READS: The average frequency of preparation experiences with Maintaining a Positive Classroom Climate for teachers in 
the BA program in Provider 1 is 4.1. Teacher responses varied from 2.4 to 5.0. 

SOURCE: Study’s teacher survey data, 2015. 


2. Degree program level: Percentage of the total variance that was explained by degree programs 
within preparation providers. The larger this percentage was, the more varied the preparation 
experiences were between degree programs within preparation providers. 


3. Teacher level: Percentage of the total variance that was explained by teachers within degree 
programs within preparation providers. The larger this percentage was, the more varied the 
preparation experiences were across teachers within the same degree program and preparation 
provider. 
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For all preparation experiences, we found a pattern of more variation among 
teachers from programs by the same provider than between teachers from 
different program providers. 


For example, for the 13 competency areas, a minimum of 86 percent of the variance in preparation 
experiences was at the level of individual teachers. With the exception of the Practice learning 
opportunity, individual teachers accounted for nearly 90 percent of the variance in types of learning 
opportunities. For Practice, a somewhat smaller portion of the variance (82 percent) was at the level of 
individual teachers. (See table J.1.) 


Table J.1: Decomposition of the Total Variance in the Average Frequency of Preparation 
Experiences With Competency Areas and Through Types of Learning Opportunities 


Percentage of Total Variance at the... 
Preparation Degree 
Provider Level Program Level Teacher Level 

Competency Area 

Maintaining a Positive Classroom Climate A% 10% 86% 
Managing Student Behavior to Maximize Learning Time 2% 5% 94% 
Productive Use of Classroom Time 4% 6% 90% 
Conveying the Importance of Learning 1% 6% 93% 
Demonstrating Sensitivity to Student Needs 1% 5% 88% 
Designing and Using Assessments of Student Learning 3% 5% 92% 
Effective English Language Arts Instruction 4% 6% 90% 
Building Students’ Higher-Order Thinking Skills 3% 6% 91% 
Facilitating Extended Classroom Discussions 1% 9% 90% 
Providing Feedback That Helps Student Learning 4% 5% 91% 
Effective Mathematics Instruction 2% 71% 91% 
Building Comprehension of Academic Concepts 6% 5% 89% 
Effective Instruction for English Learners 6% 5% 88% 
Type of Learning Opportunity 

Coursework 3% 4% 93% 
Observation 3% 8% 89% 
Practice 6% 11% 83% 
Feedback 4% 5% 91% 


NOTE: Teachers’ responses to the survey questions about preparation experiences ranged from 1 to 5, where 1 = “Rarely/Never” 
and 5 = “Very often.” Three observations were excluded from this analysis due to missing data on degree program. The sample was 
restricted to preparation providers that were represented by five or more teachers. Sample size varied between 2,572 and 2,598. 
Columns may not equal 100 percent across rows due to rounding. 

TABLE READS: Preparation providers accounted for 4 percent of the variation in teachers’ overall average preparation experiences; 
the specific degree programs teachers attended within the preparation providers accounted for an additional 9 percent. The 
remaining variance (86 percent) was estimated to be at the teacher level. 

SOURCE: Study’s teacher survey data, 2015. 
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Appendix K. Robustness of Results to Alternative Statistical Models 


The study’s primary models of the relationships of preparation experiences to teaching effectiveness 
include indicators for the sites (state or school districts)'’ from which the measures of effectiveness were 
estimated, and controls for whether teachers were in their first, second, or third year of teaching at the 
time their effectiveness was measured. The site indicators were a crucial part of the models because the 
measures of effectiveness are relative to other teachers within the site. This means that an average teacher 
in Site A is more effective than a below-average teacher in Site A, and an average teacher in Site B is 
more effective than a below-average teacher in Site B, but that the models do not tell us whether an 
average teacher at Site A is more, less, or equally as effective as an average teacher in Site B. 


By including indicator variables for sites as covariates in the models, we ensure that the relationships 
between preparation experiences and effectiveness are estimated within sites, where relative effectiveness 
is measured, and then that the within-site relationships are averaged across sites to give an overall, 
average measure of the relationships between preparation experiences and effectiveness. The primary 
models also control for years of teaching experience because tenure was found to have a strong 
relationship to effectiveness in our data and has been consistently shown to be related to teaching 
effectiveness in other studies. '® 


Other characteristics of teachers and their classes, however, may be hypothesized to be related to both 
preparation experiences and teaching effectiveness, and could therefore confound the estimates of the 
relationship between the two if not accounted for either by the study design or by statistical control. For 
example, it may be hypothesized (but it has not been shown) that teacher candidates with more grit may 
seek more and find more preparation experiences, and they may be more effective teachers regardless of 
their preparation experiences. Additionally, although the measures of teaching effectiveness account for 
the demographic characteristics of the students taught, it is possible that the concentrations of students 
with particular characteristics may be related to teaching effectiveness above and beyond the effects of 
the demographics of individual students. Those aggregates of student characteristics (or measures of 
classroom composition) could also be correlated with preparation experiences if teachers with particular 
preparation experiences were more likely to be placed in settings with higher concentrations of students 
with particular demographic characteristics. If classroom characteristics were correlated with both 
preparation experiences and effectiveness, then failure to account for those classroom composition 
characteristics may confound the estimates of the relationships of preparation experiences to 
effectiveness. 


The ability of the present study to control for potential confounding factors of the types described above 
was limited by the availability of measures of those types of factors. It is therefore a limitation of the 
study that it cannot rule out competing hypotheses about why particular relationships were or were not 
observed. The lack of the ability to rule out all competing hypotheses is what makes the study an 
exploratory endeavor, as opposed to a study that can support causal inferences about relationships (e.g., a 
randomized controlled trial). Within the limitations of the study as designed, we sought to determine 


17 The student-teacher linked data were obtained from 19 separate “sites” in 14 states. Each site was a state or 
district educational agency that was able to provide the study with student-teacher linked data for all students 
and teachers in the relevant grades and years. 


18 The following studies have found years of experience to be related to effectiveness: Boyd et al. (2009), Harris 
and Sass (2007, 2011), Kini and Podolsky (2016), and Xu, Hannaway and Taylor (2009). 
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whether the estimates of relationships between preparation experiences and teaching effectiveness were 
sensitive to inclusion or exclusion of covariates representing measures of characteristics of teachers and 
classroom composition. To conduct these sensitivity analyses, we fit our primary model (Model A below) 
and three additional models with the following sets of covariates to the data: 


e Model A—included indicators for sites (districts or states) and years of teaching experience (first, 
second, or third year); 


e Model B—included Model A covariates plus aggregate means of students’ baseline achievement 
scores and demographic characteristics; 


e Model C—included Model A covariates plus measures of teacher characteristics; and 


e Model D—included Model A covariates plus aggregate means of students’ baseline achievement 
scores and demographic characteristics and measures of teacher characteristics. 


Figure K.1 presents estimates for Models A—D for creating a productive learning environment through 
Practice for ELA and math effectiveness and shows that estimates were similar whether they were 
estimated using Model A or Models B-D. For the relationships of the other preparation experience 
measures’ to ELA and math effectiveness, Models B—D also produced similar results to those obtained 
from Model A. 


19 Creating a productive learning environment through Coursework; creating a productive learning environment 


through Observation; and creating a productive learning environment through Feedback. 


NCEE 2019-4007 Teacher Preparation Experiences and Early Teaching Effectiveness 53 


Figure K.1: Using Alternative Sets of Covariates in Models to Estimate Relationships of 


Preparation Experiences to Effectiveness Produced Estimates That Were Similar to Those From 
the Primary Model 


Creating a Productive Learning Environment Through Practice 
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Covariates Included in Relational Model: 


Model A (Site indicators, year of teaching dummies) 


& Model B (Model A + Class aggregate means of pre-tests and student demographics) 


Model C (Model A + Teacher characteristics) 


Model D (Model A + Class aggregate means of pre-tests and student demographics + Teacher characteristics) 


* indicates that the coefficient was statistically different from zero at the p < .05 level. 

** indicates that the coefficient was statistically different from zero at the p < .01 level. 

NOTE: Results based on preparation experiences and effectiveness measures of 2,032 ELA and 1,894 math teachers. Class 
aggregate means of pre-tests and student demographics included measures of baseline class averages of ELA and math scores, 
percentages of students who were eligible for free or reduced-price lunch, in special education, male, Hispanic, or Asian. Teacher 
characteristics included measures of grit, prior leadership experience, age, race/ethnicity, and years of teaching at the time of the 
survey (which could be different than year of teaching at the time of the measurement of effectiveness). 

FIGURE READS: Similar results were obtained from models with different sets of covariate controls. The estimates of the 
relationship of creating a productive learning environment through Practice to effectiveness in ELA were slightly attenuated when 
additional control covariates were included in the model. Estimates of the relationship of creating a productive learning environment 
through Practice to effectiveness in math were slightly larger when additional control covariates were included in the model. 


SOURCE: Preparation experience measures: study’s teacher survey data, 2015. Effectiveness measures: study's teacher-student 
linked data, 2012-15. 
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The study investigated the relationships of preparation experiences with creating a productive learning 
environment and with promoting analytic thinking skills to teaching effectiveness for subgroups of 
teachers with differing student populations of at-risk students. Specifically, the study looks at subgroups 
of teachers who had (1) higher and lower percentages of students eligible for free or reduced-price lunch 
(FRPL), and (2) classrooms in which average baseline achievement was above and below the median of 
baseline achievement for the site (district or state).*° The study also looks at teaching effectiveness with a 
particular subgroup of students: English learners. 


Results Based on the Proportion of FRPL-Eligible Students 


In our sample of teachers, the percentage of students in their classrooms who were eligible for FRPL 
ranged from 0 percent to 100 percent. The mid-point break in the distribution of classroom averages for 
percentage of FRPL-eligible students was 86 percent (i.e., half of teachers had 86 percent or more FRPL- 
eligible student and half had less than 86 percent). We used this mid-point break to create two subgroups 
of teachers. We also examined the sensitivity of the findings to the choice of the average percentage used 
to create the two subgroups of teachers, by conducting parallel analyses using 50 percent and 100 percent 
FRPL-eligible as the break points. 


In these analyses, we estimated the size of the relationships between preparation experiences and 
effectiveness for each of the subgroups of teachers (those with higher and lower proportions of at-risk 
students). We estimated these relationships for both of the broad categories of preparation experiences 
and for both math and ELA teaching effectiveness, using the methodology described in appendix I. 


For the teachers with higher proportions of students eligible for FRPL there were some significant, 
positive relationships between both ELA and math teaching effectiveness and preparation experiences 
with creating a productive classroom environment (tables L.1 and L.2). No relationships were found 
between teaching effectiveness in ELA or math for preparation experiences with promoting analytic 
thinking skills for any subgroups of teachers. 


20 The student-teacher linked data were obtained from 19 separate “sites” in 14 states. Each site was a state or 


district educational agency that was able to provide the study with student-teacher linked data for all students 
and teachers in the relevant grades and years. 
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of Teachers Defined by Proportion FRPL-Eligible Students in Classrooms 


Table L.1: Relationships of Preparation Experiences to ELA Teaching Effectiveness for Subgroups 


Subgroup 
FRPL Classification of Students 
Preparation Type of Learning < 86% > 86% 
Experience Opportunity (n = 1,424) (n = 1,405) 
Pes Standard nee Standard 
Coefficient ENE Coefficient Error 
Creating a Coursework 0.020 0.046 0.056 0.048 
Productive Observation 0.038 0.049 0.100 x 0.049 
Learning Practice 0.053 0.046 0.096 + * 0.045 
Environment Feedback 0.019 0.042 0.082 0.043 
: Coursework 0.006 0.043 0.048 0.044 
ae Observation -0.015 0.044 0.038 0.043 
Thinking Skill Practice —0.010 0.044 0.064 0.043 
g Skills 
Feedback —0.002 0.039 0.058 0.039 
< 50% > 50% 
(n = 508) (n = 2,321) 
Creating a Coursework 0.032 0.073 0.040 0.037 
Productive Observation 0.031 0.080 0.076 * 0.038 
Learning Practice 0.091 0.074 0.070 * 0.036 
Environment Feedback 0.059 0.066 0.050 0.033 
; Coursework 0.024 0.072 0.022 0.034 
ae Observation 0.016 0.072 0.018 0.034 
Thinking Skills Practice 0.026 0.072 0.030 0.034 
Feedback 0.034 0.062 0.028 0.031 
< 100 % = 100% 
(n = 2,073) (n = 756) 
Creating a Coursework 0.027 0.038 0.072 0.066 
Productive Observation 0.047 0.040 0.154 = x 0.069 
Learning Practice 0.047 0.037 0.193 0.065 
Environment Feedback 0.025 0.034 0.140 = 0.060 
; Coursework 0.010 0.036 0.075 0.060 
Promoting Observation ~0.007 0.036 0.070 0.061 
rab Practice 0.002 0.035 0.107 0.060 
Thinking Skills 
Feedback 0.011 0.031 0.090 0.056 


* Coefficient for relationship of experience to effectiveness is significantly different than zero (p < .05). 

NOTE: Data from one site were omitted from this analysis because there was no variation among teachers within the site in the 
percentage of students who were FRPL eligible. For all teachers in that site, 100 percent of students were FRPL eligible. The total 
sample size for this analysis was 2,829 ELA teachers. 

TABLE READS: Among teachers whose classrooms had 86 percent or fewer FRPL students, a one unit increase in experience in 
creating a productive learning environment obtained through coursework was associated with a 0.020 standard deviation unit 
increase in teaching effectiveness. This association was not significantly different than zero. 

SOURCE: Preparation experience measures: study’s teacher survey data, 2015. Effectiveness measures: study’s teacher/student 
linked data, 2012-15. 
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Table L.2. Relationships of Preparation Experiences to Math Teaching Effectiveness for 
Subgroups of Teachers Defined by Proportion FRPL-Eligible Students in Classrooms 


Subgroup 
7 FRPL Classification of Students 
Preparation ne < 86% > 86% 
Experience Learning Opportunity (n= al ee (n= ater 
Coefficient Error Coefficient Error 
Creating a Coursework -0.062 0.044 0.082 0.045 
Productive Observation -0.013 0.047 0.112 = 0.047 
Learning Practice 0.012 0.042 0.109 =x 0.044 
Environment Feedback 0.004 0.040 0.053 0.041 
; Coursework -0.010 0.040 0.004 0.040 
pen Observation ~0.036 0.041 0.039 0.041 
is Practice -0.027 0.041 0.056 0.040 
Thinking Skills 
Feedback -0.024 0.037 0.030 0.037 
< 50% > 50% 
(n = 470) (n = 2,177) 
Creating a Coursework 0.021 0.070 0.006 0.036 
Productive Observation 0.053 0.075 0.050 0.037 
Learning Practice 0.093 0.068 0.050 0.034 
Environment Feedback 0.111 0.066 0.007 0.032 
; Coursework 0.029 0.068 0.011 0.031 
paren Observation 0.044 0.068 -0.005 0.032 
Thinking Skil Practice 0.077 0.069 0.003 0.032 
g Skills 
Feedback 0.069 0.061 -0.012 0.029 
< 100% = 100% 
(n = 1,945) (n = 702) 
Creating a Coursework -0.021 0.037 0.097 0.062 
Productive Observation 0.024 0.038 0.128 0.066 
Learning Practice 0.052 0.035 0.086 0.061 
Environment Feedback 0.014 0.033 0.068 0.056 
‘ Coursework 0.003 0.033 —0.000 0.055 
Promoting Observation ~0.002 0.034 0.018 0.055 
ee ; Practice 0.022 0.034 0.003 0.055 
inking Skills 
Feedback 0.004 0.030 0.006 0.051 


* Coefficient for relationship of experience to effectiveness is significantly different than zero (p < .05). 

NOTE: Data from one site were omitted from this analysis because there was no variation among teachers within the site in the 
percentage of students who were FRPL eligible. For all teachers in that site, 100 percent of students were FRPL eligible. The total 
sample size for this analysis was 2,647 math teachers. 

TABLE READS: Among teachers whose classrooms had 86 percent or fewer FRPL students, a one unit increase in experience in 
creating a productive learning environment obtained through coursework was associated with a 0.062 standard deviation unit 
decrease in teaching effectiveness. This association was not significantly different than zero. 

SOURCE: Preparation experience measures: study’s teacher survey data, 2015. Effectiveness measures: study’s teacher/student 
linked data, 2012-15. 


Results Based on Classroom Average Baseline Student Academic Achievement 


We also created subgroups of teachers based on the average baseline ELA or math achievement scores of 
the students in teachers’ classrooms. We created two subgroups of teachers: (1) teachers whose students, 
on average, were in the bottom 50 percent of the distribution; and (2) teachers whose students were in the 
top 50 percent of the distribution. As with the analyses described above, we estimated the size of the 
relationships between preparation experiences and effectiveness for each subgroup of teachers for both of 
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the broad categories of preparation experiences and for both math and ELA teaching effectiveness (see 
table L.3). 


For teachers with students at greater risk, there was a statistically significant relationship between 
experiences with creating a productive learning environment through Practice and ELA teaching 
effectiveness. No other relationships were found with teaching effectiveness in ELA or math. 


Table L.3: Relationships of Preparation Experiences to Teaching Effectiveness in ELA and Math 
for Subgroups of Teachers Defined by Class Average Baseline ELA or Math Performance 


Subgroup 
= Baseline Performance Classification of Students 
Preparation ds Top Half Bottom Half 
Experience Learning Opportunity (n= +e (n= 1) __ 
Coefficient Error Coefficient Error 

Creating a Coursework 0.062 0.044 0.019 0.050 

Productive Observation 0.072 0.046 0.070 0.050 

Learning Practice 0.093 * 0.044 0.063 0.046 

< Environment Feedback 0.047 0.040 0.070 0.043 

ul Coursework 0.037 0.040 0.008 0.046 

nae a Observation 0.035 0.041 ~0.001 0.045 

Skills Practice 0.041 0.041 0.022 0.044 

Feedback 0.038 0.037 0.030 0.040 

Top Half Bottom Half 
(n = 1,365) (n = 1,369) 

Creating a Coursework 0.014 0.043 -0.001 0.047 

Productive Observation 0.074 0.045 0.019 0.049 

Learning Practice 0.079 0.043 0.044 0.043 

= | Environment Feedback 0.038 0.040 0.021 0.041 

= Coursework 0.007 0.039 -0.031 0.041 

Promenng) Observation ~0.003 0.040 |  -0.002 0.041 
Analytic Thinking 

Skills Practice 0.008 0.040 0.018 0.041 

Feedback 0.004 0.036 0.000 0.037 


* Coefficient for relationship of preparation experiences to teaching effectiveness is significantly different than zero (p < .05). 
TABLE READS: Among teachers whose classroom average baseline scores were in the top half of the distribution, a one unit 
increase in preparation experiences in creating a productive learning environment obtained through Coursework was associated 
with a 0.062 standard deviation unit increase in ELA teaching effectiveness. This association was not significantly different than 
zero. 

SOURCE: Preparation experience measures: study’s teacher survey data, 2015. Effectiveness measures: study’s teacher/student 
linked data, 2012-15. 


Results for Teachers of English Language Learners 


We examined the relationships between teachers’ preparation experiences and their effectiveness with 
English language learners. These analyses were based on the subset of just over a third of the sample 
teachers for whom we had test score data and who had at least one English learner in their classroom 
(1,146 ELA and 1,062 math teachers). We used the same value-added methodology that we used for 
creating the effectiveness scores with all students (see appendix J), but included only the test scores of 
English language learners in the value-added model. 
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There were no significant relationships between the two broad categories of preparation experiences 
measured in this study and teaching effectiveness with English learners, for either math or ELA (see 
figure L.1). There were also no significant relationships between preparation experiences in the 
competency area Effective Instruction for English Learners and teaching effectiveness with English 
learners, for either ELA or math (see table L.4). 


Figure L.1: Relationships of Preparation Experiences With Creating a Productive Learning 
Environment or With Promoting Analytic Thinking Skills to ELA and Math Teaching 
Effectiveness for English Learners 


Creating a Productive Learning Environment Promoting Analytic Thinking Skills 
Coursework ++o— Coeff = 0.042 Coursework H—-@— Coeff = 0.080 
<i Observation 4H—-@— Coeff = 0.078 = Observation H-@— Coeff = 0.064 
Practice Le Coeff = 0.089 Practice Leo Coeff = 0.084 
Feedback to Coeff = 0.032 Feedback e— Coeff = 0.039 
Coursework a Coeff = -0.036 Coursework = Coeff = -0.040 
s Observation rH Coeff = 0.052 ts Observation aan Coeff = 0.004 
Practice +i Coeff = 0.046 Practice tii Coeff = 0.027 
Feedback fi Coeff = 0.018 Feedback a_i Coeff= 0.015 
05 03 04 o1 03 05 05 03 -01 0.1 0.3 05 
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NOTE: Results for EL students are based on preparation experiences and effectiveness measures of 1,146 ELA and 1,062 math 
teachers. 

FIGURE READS: The coefficients summarizing the relationships of preparation experiences with creating a productive learning 
environment and promoting analytic thinking skills obtained through Coursework to effectiveness in ELA for EL students were 0.042 
and 0.080, respectively. They were not statistically significant, as indicated by the confidence interval crossing zero. 

SOURCE: Preparation experience measures: study’s teacher survey data, 2015. Effectiveness measures: study’s teacher/student 
linked data, 2012-15. 


Table L.4: Standardized Regression Coefficients for Relationships of Preparation Experiences 
With Effective Instruction for English Learners to ELA and Math Teaching Effectiveness for 
English Learners 


Effective Instruction for English Learners by Type of Learning Opportunity 

Coursework Observation Practice Feedback 
nai 146 teachers) 0.033 0.015 0.023 0.008 
aa 062 teachers) -0.015 0.010 0.005 0.008 


NOTE: None of the coefficients is statistically significant. 

FIGURE READS: The regression coefficient for the relationship of preparation experiences with Effective Instruction for English 
Learners obtained through Coursework to ELA teaching effectiveness for English learners is 0.033, but is not statistically 
significantly greater than zero. The relationship was estimated using data from n = 1,146 ELA teachers of EL students. 

SOURCE: Preparation experience measures: study’s teacher survey data, 2015. Effectiveness measures: study’s teacher/student 
linked data, 2012-15. 
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Appendix M. Relationships Between Preparation Experiences and 


Effectiveness by Year of Teaching 


This appendix summarizes results of analyses to test whether the magnitudes of the relationships of 
preparation experiences to effectiveness differed for teachers in their first, second, and third years of 
teaching. The tests were conducted in models that included interaction terms between the preparation 
experience measure and the year of teaching. The p-values are from F-tests of the null hypothesis that the 
relationship between preparation experiences and effectiveness does not differ between first, second, and 
third year teachers. A significant result (p < .05) would indicate that the relationships vary by year of 
teaching experience. Results are presented for 16 separate analyses corresponding to: 


e Relationships between preparation experiences in creating a productive learning environment 
and effectiveness in ELA by year of teaching, for each of the four types of learning opportunities 
(table M.1). 

e Relationships between preparation experiences in creating a productive learning environment 
and effectiveness in math by year of teaching, for each of the four types of learning opportunities 
(table M.2). 

e Relationships between preparation experiences in promoting analytic thinking skills and 
effectiveness in ELA by year of teaching, for each of the four types of learning opportunities 
(table M.3). 

e Relationships between preparation experiences in promoting analytic thinking skills and 
effectiveness in math by year of teaching, for each of the four types of learning opportunities 
(table M.4). 


The results of all 16 analyses indicated no significant differences in the relationship between preparation 
experiences and effectiveness by year of teaching. 


Table M.1: Standardized Regression Coefficients for Relationships of Preparation Experiences 
With Creating a Productive Learning Environment to ELA Teaching Effectiveness by Year of 
Teaching 


Type of Learning Year of Teaching 
Opportunity 1st Year 2nd Year 34 Year p-value 
Coursework 0.03 0.07 0.06 0.89 
Observation 0.06 0.11 0.09 0.72 
Practice 0.09 0.10 0.10 0.95 
Feedback 0.05 0.10 0.09 0.68 


p-Value is for test of whether relationship between preparation experience and teaching effectiveness differs by year of teaching. 
FIGURE READS: The increase in ELA teaching effectiveness associated with a one-unit increase in creating a productive learning 
environment through Coursework (on a five-point scale ranging from 1 = “Rarely/Never” to 5 = “Very often”) was 0.03 standard 
deviation (SD) units for first-year teachers, 0.07 SD units for second-year teachers, and 0.06 SD units for third-year teachers. There 
were no Statistically significant differences in the relationships by year of teaching. 

SOURCE: Preparation experience measures: study's teacher survey data, 2015. Effectiveness measures: study’s teacher/student 
linked data, 2012-15. 
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Table M.2: Standardized Regression Coefficients for Relationships of Preparation Experiences 


With Creating a Productive Learning Environment to Math Teaching Effectiveness by Year of 
Teaching 


Type of Learning Year of Teaching 
Opportunity 1st Year 2nd Year 34 Year p-value 
Coursework -0.01 0.00 0.08 0.60 
Observation 0.04 0.05 0.11 0.95 
Practice 0.04 0.08 0.11 0.74 
Feedback 0.00 0.07 0.10 0.54 


p-Value is for test of whether relationship between preparation experience and teaching effectiveness differs by year of teaching. 
FIGURE READS: The difference in math teaching effectiveness associated with a one-unit increase in creating a productive 
learning environment through Coursework (on a five-point scale ranging from 1 = “Rarely/Never” to 5 = “Very often”) was —0.01 
standard deviation (SD) units for first-year teachers, 0.00 SD units for second-year teachers, and 0.08 SD units for third-year 
teachers. There were no statistically significant differences in the relationships by year of teaching. 


SOURCE: Preparation experience measures: study’s teacher survey data, 2015. Effectiveness measures: study’s teacher/student 
linked data, 2012-15. 


Table M.3: Standardized Regression Coefficients for Relationships of Preparation Experiences 
With Promoting Analytic Thinking Skills to ELA Teaching Effectiveness by Year of Teaching 


Type of Learning Year of Teaching 
Opportunity 1st Year 2nd Year 34 Year p-value 
Coursework 0.03 0.03 0.04 0.95 
Observation 0.03 0.03 0.02 0.89 
Practice 0.06 0.03 0.02 0.82 
Feedback 0.05 0.05 0.06 0.90 


p-Value is for test of whether relationship between preparation experience and teaching effectiveness differs by year of teaching. 
FIGURE READS: The increase in ELA teaching effectiveness associated with a one-unit increase in promoting analytic thinking 
skills through Coursework (on a five-point scale ranging from 1 = “Rarely/Never” to 5 = “Very often”) was 0.03 standard deviation 
(SD) units for first-year teachers, 0.03 SD units for second-year teachers, and 0.04 SD units for third-year teachers. There were no 
statistically significant differences in the relationships by year of teaching. 


SOURCE: Preparation experience measures: study’s teacher survey data, 2015. Effectiveness measures: study’s teacher/student 
linked data, 2012-15. 


Table M.4: Standardized Regression Coefficients for Relationships of Preparation Experiences 
With Promoting Analytic Thinking Skills to Math Teaching Effectiveness by Year of Teaching 


Type of Learning Year of Teaching 
Opportunity 1st Year 2nd Year 34 Year p-value 
Coursework -0.03 -0.03 0.11 0.37 
Observation -0.02 0.01 0.10 0.74 
Practice -0.02 0.03 0.11 0.51 
Feedback -0.02 0.01 0.09 0.69 


p-Value is for test of whether relationship between preparation experience and teaching effectiveness differs by year of teaching. 
FIGURE READS: The difference in math teaching effectiveness associated with a one-unit increase in promoting analytic thinking 
skills through Coursework (on a five-point scale ranging from 1 = “Rarely/Never” to 5 = “Very often”) was -0.03 standard deviation 
(SD) units for first-year teachers, -0.03 SD units for second-year teachers, and 0.11 SD units for third-year teachers. There were no 
statistically significant differences in the relationships by year of teaching. 


SOURCE: Preparation experience measures: study’s teacher survey data, 2015. Effectiveness measures: study’s teacher/student 
linked data, 2012-15. 
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